
Calhoun 

iniQiuiic^iul Ar{hiv« of tilt Mil vdl Poii^roduiit School 


Calhoun: The NPS Institutional Archive 
□Space Repository 



Theses and Dissertations 


1. Thesis and Dissertation Collection, all items 


2010-09 

Applicability of performance assessment tools 
to Marine Corps Air Ground Task Force C4 
System of Systems Performance Assessment 

O'Neil, Michael S. 

Monterey, California. Naval Postgraduate School 
http://hdl.handle.net/10945/5146 

This publication is a work of the U.S. Government as defined in Title 17, United 
States Code, Section 101. Copyright protection is not available for this work in the 
United States. 

Downloaded from NPS Archive: Calhoun 



DUDLEY 

KNOX 

LIBRARY 


htt p://w ww. n ps. e du/l ib ra ry 


Caflwuo is the Naval Postgraduate School's public access digital repository for 
research mate rials and institutiional publicatkios created by the NPS community. 
Calhoun is named for Professor of Mathematics Guy K. CaSiouo, NPS's first 
appointed — and published — schofaily author. 

Dudley Knox Library / Naval Postgraduate School 
411 Dyer Road / 1 Univefsity Circle 
Monterey, California USA 93943 







NAVAL 

POSTGRADUATE 

SCHOOL 

MONTEREY, CALIFORNIA 


THESIS 


APPLICABILITY OF PERFORMANCE ASSESSMENT TOOLS 
TO MARINE CORPS AIR GROUND TASK FORCE C4 SYSTEM 
OF SYSTEMS PERFORMANCE ASSESSMENT 

by 

Michael O’Neil 
September 2010 

Thesis Advisor; Gregory Miller 

Second Reader: David Rathgeber 


Approved for public release; distribution is unlimited 




THIS PAGE INTENTIONALLY LEET BLANK 



1 REPORT DOCUMENTATION PAGE 

Form Approved 0MB No. 0704-0188 \ 

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instruction, 
searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send 
comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to 
Washington headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 
22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188) Washington DC 20503. 

1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED 

September 2010 Master’s Thesis 

4. TITLE AND SUBTITLE Applicability of Performance Assessment Tools to 
Marine Corps Air Ground Task Force C4 System of Systems Performance 

Assessment 

5. EUNDING NUMBERS 

6. AUTHOR(S) Michael S. O’Neil 

7. PEREORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

Naval Postgraduate School 

Monterey, CA 93943-5000 

8. PEREORMING ORGANIZATION 
REPORT NUMBER 

9. SPONSORING /MONITORING AGENCY NAME(S) AND ADDRESS(ES) 

Marine Corps Tactical Systems Support Activity 

Camp Pendleton, CA 92055 

10. SPONSORING/MONITORING 
AGENCY REPORT NUMBER 

II. SUPPLEMENTARY NOTES The views expressed in this thesis are those of the author and do not reflect the official policy 
or position of the Department of Defense or the U.S. Government. IRB Protocol number 

I2a. DISTRIBUTION / AVAILABILITY STATEMENT 

Approved for public release; distribution unlimited 

12b. DISTRIBUTION CODE 

A 

13. ABSTRACT (maximum 200 words) 

This research focuses on the application of existing assessment tools that may be applicable to Marine Air Ground 
Task Force (MAGTF) Command, Control, Communications and Computers (C4) System of Systems (SoS) 
performance assessment efforts. An analysis of the Marine Corps Tactical Systems Support Activity’s (MCTSSA’s) 
C4 SoS assessment approach provides a means for defining a MAGTF C4 SoS and for illustrating how that SoS is 
represented in the assessment environment. This provides the framework and context for follow-on examination of 
SoS performance metrics. The challenges with defining specific performance metrics and examination of past 
assessment events using those metrics provide the basis for discussion of alternative approaches and application of 
assessment tools specifically tailored for SoS assessment efforts. Three specific tools, i-Score, Interoperability 
Quotient (IQ), and Dynamic Software Architecture Visualization and Evaluation (DynSAVE), are examined. The 
results indicate i-Score and DynSAVE offer the greatest potential applicability to the MAGTF SoS assessment effort. 
In a culminating discussion, applying a multicriteria identification process to obtain a mathematical model that 
correlates an interoperability measure with measurable SoS performance criteria is proposed as a means of extending 
the i-Score model for greater applicability to SoS assessment and performance improvement efforts. 

14. SUBJECT TERMS SoS, Interoperability, IQ, SAVE, i-Score, MAGTF 

15. NUMBER OE 

PAGES 

93 

16. PRICE CODE 

17. SECURITY 
CLASSIEICATION OE 
REPORT 

Unclassified 

18. SECURITY 

CLASSIEICATION OE THIS 
PAGE 

Unclassified 

19. SECURITY 
CLASSIEICATION OE 
ABSTRACT 

Unclassified 

20. LIMITATION OE 
ABSTRACT 

UU 


NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) 


Prescribed by ANSI Std. 239-18 


1 




























THIS PAGE INTENTIONALLY LEET BLANK 


11 



Approved for public release; distribution is unlimited 


APPLICABILITY OF PERFORMANCE ASSESSMENT TOOLS TO MARINE 
CORPS AIR GROUND TASK FORCE C4 SYSTEM OF SYSTEMS 
PERFORMANCE ASSESSMENT 


Michael S. O’Neil 

Major (Retired), United States Marine Corps 
B.S., Virginia Military Institute, 1980 
M.S., Naval Postgraduate School, 1993 


Submitted in partial fulfillment of the 
requirements for the degree of 


MASTER OF SCIENCE IN SYSTEMS ENGINEERING MANAGEMENT 


from the 


NAVAL POSTGRADUATE SCHOOL 
September 2010 


Author: Miehael O’Neil 


Approved by: Gregory Miller 

Thesis Advisor 


Dr. David Rathgeber 
Second Reader 


Dr. Clifford Whitcomb 

Chairman, Department of Systems Engineering 



THIS PAGE INTENTIONALLY LEET BLANK 


IV 



ABSTRACT 


This research focuses on the application of existing assessment tools that may be 
applicable to Marine Air Ground Task Force (MAGTF) Command, Control, 
Communications and Computers (C4) System of Systems (SoS) performance assessment 
efforts. An analysis of the Marine Corps Tactical Systems Support Activity’s 
(MCTSSA’s) C4 SoS assessment approach provides a means for defining a MAGTF C4 
SoS and for illustrating how that SoS is represented in the assessment environment. This 
provides the framework and context for follow-on examination of SoS performance 
metrics. The challenges with defining specific performance metrics and examination of 
past assessment events using those metrics provide the basis for discussion of alternative 
approaches and application of assessment tools specifically tailored for SoS assessment 
efforts. Three specific tools, i-Score, Interoperability Quotient (IQ), and Dynamic 
Software Architecture Visualization and Evaluation (DynSAVE), are examined. The 
results indicate i-Score and DynSAVE offer the greatest potential applicability to the 
MAGTE SoS assessment effort. In a culminating discussion, applying a multicriteria 
identification process to obtain a mathematical model that correlates an interoperability 
measure with measurable SoS performance criteria is proposed as a means of extending 
the i-Score model for greater applicability to SoS assessment and performance 
improvement efforts. 
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EXECUTIVE SUMMARY 


Over a number of years, the Marine Corps Taetical Systems Support Activity has 
aggressively pursued new and innovative ways to improve efforts to assess the 
performance of tactical C4 Systems of Systems prior to deployment in an operational 
environment. In that pursuit, determining how best to approach performance assessments 
of large-scale, complex SoS has proven a challenge that extends beyond the Marine 
Corps to a fundamental system of systems engineering challenge shared by acquisition 
and test organizations and activities throughout the Department of Defense. Selecting 
key performance criteria, developing methodologies for conducting large-scale SoS 
assessments and defining more quantitative means for measuring and assessing SoS 
performance and behavior all serve as a central challenge and the genesis of this study. 

After a high-level review of efforts to address the challenges associated with SoS 
assessments at both Joint and Marine Corps component level, three tools were examined 
for their applicability specifically to C4 SoS assessment efforts. While each tool provides 
a unique approach that may help better quantify the performance of a MAGTF C4 SoS, 
they are also very similar in that they each examine information exchanges at the 
component level to derive an overall SoS performance assessment measure in terms of 
interoperability. The tools examined included the i-Score methodology developed 
through the Air Force Institute of Technology, the Interoperability Quotient (IQ) 
methodology developed by Northrop Grumman Corporation and the Dynamic Software 
Architecture Visualization and Evaluation (DynSAVE) tool and process developed by the 
Eraunhofer Center for Experimental Software Engineering Maryland. 

Of these tools, i-Score and DynSAVE offer the most potential for follow-on 
analysis, with the i-Score methodology providing a means for deriving a quantitative 
measure of SoS interoperability and the DynSAVE software tool providing a means of 
quantifying the anomalous behavior and relative efficiency of the SoS. A discussion of 
an approach to develop a mathematical model that correlates an interoperability measure 
with measurable SoS performance criteria using a multicriteria identification process 

serves as the culminating effort of this study. With a mature mathematical model and 
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high degree of eonfidenee in the eorrelation between the performanee eriteria measure 
predieted by the interoperability model and the performanee measure observed from the 
prototype, the mathematieal model may provide a eost effective and rapid means to 
examine the SoS for potential performance improvement. 



LIST OF ACRONYMS AND ABBREVIATIONS 


-A- 

ACE.Aviation Combat Element 

AFIT.Air Force Institute of 

Technology 

AFATDS.Advanced Field Artillery 

Tactical Data System 

AT&L.Acquisition Technology & 

Logistics 


-C- 

C2.Command and Control 

C4.Command, Control, 

Communications, and 
Computers 

CAC2S.Common Aviation Command 

and Control System 

CE.Command Element 

COC.Combat Operations Center 

CPOF.Command Post of the Future 

CTM.Capability Test Methodology 

-D- 

DAG.Defense Acquisition Guide 

DASC.Direct Air Support Center 

DAU.Defense Acquisition 

University 

DIS.Distributed Interactive 

Simulation 

DoD.Department of Defense 

DOT&E.Director Operational Test & 

Evaluation 

DynSAVE.Dynamic Software 

Architecture Visualization 
and Evaluation 


-F- 

FAC.Forward Air Controller 

FC-MD.Fraunhofer Center for 

Experimental Software 
Engineering Maryland 

FCS.Future Combat Systems 

FEDOS.Federation of Systems 

FOC.Full Operational Capability 

FSCC.Fire Support Coordination 

Center 


-G- 

GCE.Ground Combat Element 

GCSS.Global Combat Support 

System 


-H- 

FICI.Fluman Computer Interface 

FILA.Fligh-Level Architecture 


-I- 

lA.Information Assurance 

ICAS.Immediate Close Air Support 

ICD.Interface Control Document 

InterTEC.Interoperability Test and 

Evaluation Capability 

IOC.Initial Operational Capability 

low.Intelligence Operations 

Workstation 

IQ.Interoperability Quotient 

-J- 

JADOCS.Joint Automated Deep 

Operations Coordination 
System 

JBD2.Joint Battlespace Dynamic 

Deconfliction 

JCIDS.Joint Capabilities Integration 

and Development System 

JITC.Joint Interoperability Test 

Command 

JMETC.Joint Mission Environment 

Test Capability 

JMT.Joint Mission Threads 

JNMS.Joint Network Management 

Systems 

JTEM.Joint Test and Evaluation 

Methodology 

JTEN.Joint Training and 

Experimentation Network 


-K- 

KPP.Key Performance Parameter 


XV 
















































-L- 

LCIM.Levels of Conceptual 

Interoperability Model 

LCE.Logistics Combat Element 

LISI.Level of Information System 

Interoperability 

LVC.Live, Virtual and Constructive 


-M- 

MACCS.Marine Air Command and 

Control System 

MAGTF.Marine Air Ground Task Force 

M&S.Modeling and Simulation 

MCSC.Marine Corps Systems 

Command 

MC3.MAGTF C4I Capabilities 

Certification 

MCOTEA.Marine Corps Operational Test 

and Evaluation Activity 

MCTSSA.Marine Corps Tactical Systems 

Support Activity 

MEF.Marine Expeditionary Force 

MEU.Marine Expeditionary Unit 

-N- 

NC3TA.NATO C3 Technical 

Architecture 


-O- 

OIM.Organizational Interoperability 

Maturity Model 

OSD.Office of the Secretary of 

Defense 

ONR.Office of Naval Research 

-P- 

POR.Program of Record 


PPBE.Planning, Programming, 

Budget and Execution 


-R- 

R&D.Research and Development 


-S- 

SAVE.Software Architecture 

Visualization and 
Evaluation 

SDREN.Secure Defense Research and 

Engineering Network 

SE.Systems Engineering 

SIAT.Systems Engineering, 

Interoperability, 
Architectures & 
Technology 

SOA.Service Oriented Architecture 

SoS.System of Systems 

SOSI.System of Systems 

Interoperability 

SPAWAR.Space and Naval Warfare 

Systems Command 


-T- 

T&E.Test and Evaluation 

TDN.Tactical Data Network 

TENA.Test and Training Enabling 

Architecture 

TLDFIS.Target Location, Designation 

and Fland-off System 

TTP.Tactics, Techniques and 

Procedures 


-U- 

UOC.Unit Operations Center 


XVI 





































I. 


INTRODUCTION 


A. BACKGROUND 

In warfare past and present, the pursuit of improved and more effective Command 
and Control (C2) on the battlefield has been, and is still, a relentless effort. As one of the 
Marine Corps’ fundamental warfighting functions, C2 is unique in that it enables all the 
other warfighting functions: intelligence, maneuver, logistics, fires and force protection 
(U.S. Marine Corps, 2001). Through C2, these functions are executed with a coherency 
and purpose that results in an overall warfighting capability that is greater than the sum of 
the capabilities provided by the individual warfighting functions. 

The proliferation of modern Command, Control, Communications and Computer 
(C4) systems and C4 system of systems (SoS) on the battlefield now serves as the focus 
of today’s effort to improve the effectiveness of C2. Advancements in technology and 
interoperability pursued and leveraged through C4 systems development efforts have 
contributed to unprecedented C2 capabilities, allowing once disparate systems to work 
together in a collaborative and synergistic manner that significantly enhances the ability 
of a commander to manage the battlespace. 

However, at the same time, the advent of C4 systems has added a new dimension 
of complexity to the battlefield. The C4 systems and SoS are now an integral part of the 
battlespace, with the performance of the C4 SoS very much interconnected to the 
successful execution of the warfighting functions in pursuit of battlespace domination. 

Furthermore, the dynamics of C4 SoS performance are in themselves complex 
and multidimensional. Changes to an individual subsystem in the SoS architecture may 
result in second- and third-order effects that result in degradation to the performance of 
the overall SoS that may be unrelated to the performance of the subsystem that has 
undergone change. To understand these complexities and to ensure that a modification to 
an existing C4 SoS architecture does not degrade overall performance of the SoS, the 
Marine Corps, for a number of years, has attempted to establish processes and procedures 
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for conducting C4 SoS assessment testing. However, while methodologies for testing 
individual systems are well doeumented and understood, there are a number of unique 
ehallenges assoeiated with SoS assessment testing. 

One of the most basie and eritieal ehallenges is determining appropriate testable 
performanee measures for the SoS. While the individual systems that make up the SoS 
typieally have well doeumented performanee eriteria that ean be used as the basis for an 
assessment, in many eases the SoS may not. The ehallenge with determining appropriate 
performanee measures in eoneert with many other eomplexities assoeiated with SoS 
testing has resulted in testing approaehes that differ very little from the methodology used 
to test the individual systems that eonstitute the SoS. That is to say, from a measurement 
and performanee perspeetive, the individual systems in the SoS tend to serve as the foeus 
of the test, whieh may not neeessarily provide a eomplete or useful assessment of the SoS 
as a whole. This seems intuitive from the definition of a SoS “as a set or arrangement of 
systems that results from independent systems integrated into a larger system that delivers 
unique capabilities” (DAU, 2010). Those unique SoS eapabilities imply unique attributes 
and behaviors derived from the arrangement and integration of the independent systems. 
These eombined eapabilities, attributes and behaviors uniquely eharaeterize the SoS and 
imply that performanee measures beyond the performanee measures assoeiated with the 
independent systems are needed to assess the SoS. 

B. PURPOSE 

In the Offiee of the Under Seeretary of Defense (Aequisition, Teehnology and 
Logisties) publieation. Systems Engineering Guide for Systems of Systems (2008), a 
number of eonditions are deseribed that eontribute to the ehallenges assoeiated with 
Systems of Systems engineering. With Systems of Systems often eomprised of 
individual systems in different stages of a very dynamie and often asynehronous 
aequisition life eyele management proeess and erossing many organizational 
management boundaries within the DoD and eommereial industry, there is a need to 
expand and redefine systems engineering proeesses to aoeommodate these needs (Offiee 
of the Deputy Under Seeretary of Defense for Aequisition and Teehnology [OSD], 2008): 
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Beyond these development ehallenges, depending on the eomplexity and 
distribution of the eonstituent systems, it may be infeasible or very 
diffieult to eompletely test and evaluate SoS eapabilities. (OSD, 2008, 
p. 13) 

While the Department of Defense (DoD) and systems engineering eommunity 
eontinue to refine and mature the proeesses for SoS engineering and development, the 
systems engineering testing eommunity has struggled with defining how best to measure 
and assess the performanee of the SoS. From that struggle, various test tools and test 
methodologies have been developed and demonstrated to address the test and evaluation 
ehallenge. This thesis investigates a small subset of existing assessment tools and 
analysis eoneepts that may be extended to the Marine Corps’ C4 System of Systems 
assessment methodology as a means to obtain a more holistie assessment of the SoS 
under evaluation. 

C. RESEARCH QUESTIONS 

Test and Evaluation (T&E) activities span the continuum of a disciplined systems 
engineering approach. As the engineering community attempts to understand and deal 
with the complexities of large-scale system of systems development efforts, the need to 
apply, adapt and modify T&E activities to accommodate a SoS engineering approach are 
evident. The following questions were designed to understand the challenges with SoS 
testing and opportunities for addressing those challenges through specific tools and 
analysis processes. While the questions are generic in nature, the intent is to address 
them in context with the specific challenges associated with the Marine Corps’ approach 
to SoS assessment testing. 

1. How can large-scale C4 System of Systems be tested and evaluated in a 
manner that reflects performance attributes associated with the System of 
Systems as a whole? 

2. What are the key attributes of a C4 SoS that may serve as the basis for SoS 
level performance criteria? 

3. What are some existing assessment tools and how may they be extended to aid 
in C4 SoS assessment process? 
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4. How might multicriteria identification improve or contribute to the Marine 
Corps’ C4 SoS assessment methodology? 

D. BENEFITS OF STUDY 

As the Marine Corps continues to refine the processes and techniques for C4 
system of systems assessment, other DoD organizations at both Joint and Component 
levels are doing the same. This thesis is intended to serve as a basis of knowledge that 
can be leveraged by the Marine Corps and other DoD test and evaluation activities, 
improving the use of assessment tools and analysis concepts to address the challenges 
associated with test and evaluation of large scale and complex C4 system of systems. 

E. SCOPE AND METHODOLOGY 

This thesis focuses on the application of existing assessment tools and analysis 
concepts that may be applicable to MAGTF C4 SoS performance assessment efforts. 
Data and documentation from past C4 SoS test events are used as well as input from the 
Marine Corps test and evaluation team planning future test events. 

An analysis of the Marine Corps Tactical Systems Support Activity’s 
(MCTSSA’s) C4 SoS assessment approach, while serving in a lead role for the Marine 
Corps’ SoS assessment effort, provides a means for defining a MAGTF C4 SoS and for 
illustrating how that SoS is represented in the assessment environment. This provides the 
framework and context for follow-on examination and discussion of performance metrics 
that are necessary for characterizing and ultimately assessing the performance of the SoS. 
The challenges with defining specific performance metrics and examination of past 
assessment events using those metrics provide the basis for discussion of alternative 
approaches and application of assessment tools specifically tailored for SoS assessment 
efforts. 

Three specific tools are examined and viewed relative to their applicability to the 
MAGTF SoS assessment effort. The tools include i-Score, an Air Force Institute of 
Technology (AFIT) initiative to measure SoS interoperability; Interoperability Quotient 
(IQ), a commercial application demonstrated by the Navy to measure SoS performance; 
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and Software Architecture Visualization and Evaluation (SAVE), a toolset developed 
through the Frau nh ofer Institute of Technology for optimizing and analyzing the 
architecture of implemented software systems. 

Building on the examination of available analysis tools, an examination of how a 
Multicriteria Identification analysis concept may be applied to a MAGTF C4 SoS and 
extended as a means to improve SoS assessment testing is investigated. This culminates 
in an overall recommendation presented in the final chapter for improving C4 SoS 
assessments through appropriate use of assessment tools and analysis concepts. Overall 
methodology is depicted in Figure 1. 



Figure 1. Study Methodology. 
Performance Tool Assessment Methodology 
Process that guided this research. 
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II. MAGTF C4 SOS TEST AND EVALUATION PROGRAM 

REVIEW 


A. INTRODUCTION 

The Marine Corps Systems Command (MCSC) serves as the Marine Corp’s agent 
for aequisition and sustainment of systems and equipment. MCTSSA is a subordinate 
command within MCSC, providing engineering and technical services in support of 
MCSC’s acquisition and sustainment role. Within the MCSC organization, MCTSSA 
falls under the cognizance of the Deputy Commander for Systems Engineering, 
Interoperability, Architectures & Technology (DC SIAT), who also serves as the Marine 
Corps’ Technical Authority Deputy Warranting Officer. SIAT is chartered with 
providing “enterprise level system engineering across product lines and product life 
cycles to ensure MCSC provides end-to-end integrated, interoperable, and certified 
warfighting capabilities” (SIAT, 2010). In effect, SIAT serves as an agent for enabling 
horizontal integration across the MCSC product groups and systems development efforts. 
The resulting goal is to bring the individual systems together into a coherent, integrated 
system of systems architecture to provide warfighting capabilities. 

In December 2006, the DC SIAT initiated an effort to develop a capability 
certification process to assess MCSC developed systems within a MAGTF SoS 
environment. MCTSSA, with a long history of supporting C4 systems throughout the 
acquisition life cycle, was tasked to develop the processes for assessing a MAGTF C4 
SoS in support of the overall certification process for the MAGTF C2 capability. 

Assessing a MAGTF C4 SoS was not new to MCTSSA. Previous attempts in the 
form of a Federation of Systems (FEDOS) test and evaluation program had met with 
mixed results. While FEDOS brought various C4 systems into a SoS environment for 
testing, no objective SoS-level performance requirements were established. With no 
established SoS-level performance requirements, the individual component systems in the 
SoS became the default focus of the test effort with performance metrics of the individual 
component systems in the SoS determined through negotiation with product sponsors. 

The product sponsors were suspect of the overall test results and questioned how the test 
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agency employed the individual system in the C4 architecture (logical and physical 
layout of the SoS test environment). Further, they failed to see value of a test that went 
beyond the scope of Developmental and Operational Test requirements as offered by 
Acosta et al. (2007); 

Further, FEDOS was perceived as a no-win situation for Product Groups. 

After a system had successfully passed Development and Operational 
Tests, and demonstrated compliance with system-level performance 
requirements as described in the Capability Development Document 
(CDD), Operational Requirements Document (ORD), or equivalent, 

FEDOS tested component systems in ways they had not been designed to 
be used. (Acosta et ah, 2007, p. 29) 

The Marine Corps’ current SoS assessment effort that resulted from the 2006 
directive from DC SIAT, termed MAGTF C4I Capabilities Certification (MC3). The 
MC3 effort is built upon the lessons learned from the EEDOS program and leverages 
efforts in the Joint test community looking to establish methods for executing tests of SoS 
in the Joint mission environment. 

B. MAGTF C4 SOS DEFINED 

Developing a common definition for a System of Systems (SoS) has been an 
evolving process and the subject of numerous discussion papers in industry and the DoD. 
In an IEEE International Conference on Systems of Systems Engineering 2009 article, 
John Clark provides a simple definition of a SoS as “The sum of the whole is greater than 
the sum of the individual parts” in which the parts are integrated and may or may not be 
members of a common domain (2009, p. 2). 

In the DoD, this definition is expanded to accommodate considerations unique to 
the warfighting domain. The basic idea of the sum of the whole being greater that sum of 
the individual parts is still present, but the definition now also infers association with a 
warfighting capability. In the Chairman of the Joint Chiefs of Staff Instruction, CJCSI 
3I70.0IE glossary, the SoS is defined as: 
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A set or arrangement of interdependent systems that are related or 
eonneeted to provide a given eapability. The loss of any part of the system 
eould signifieantly degrade the performanee or eapabilities of the whole. 

The development of an SoS solution will involve trade space between the 
systems as well as within an individual system performance. (CJCS, 2007, 
p. GL-19) 

This SoS definition is aligned with the Joint Capabilities Integration and 
Development System (JCIDS) that serves as an overarching process that guides the 
development of new capabilities for DoD Military Forces. JCIDS provides a 
methodology that is intended to improve the DoD acquisition process through a level of 
governance over the individual Service’s acquisition efforts with a focus on identifying 
shortcomings and redundancy from a Joint warfighting capabilities perspective. 

However, while JCIDS provides the acquisition community with a more holistic 
and SoS approach to the Joint acquisition process, the DoD Planning, Programming, 
Budgeting, and Execution (PPBE) process that guides acquisition funding and program 
management is not specifically structured to address the complexities of SoS acquisition 
efforts. That, along with internal acquisition community politics, legacy acquisition 
policies and many other factors has contributed to an acquisition approach that is focused 
on the individual system in terms of equipping the warfighter with a material solution 
supporting a needed capability. A system is engineered and developed to provide a 
function or capability within the context of being employed within a larger SoS (physical 
interfaces and data exchanges with other systems indentified and defined), but the SoS 
itself is more of a cooperative instantiation than an engineered entity. 

However, as the definition and understanding of the SoS concept matures, the 
engineering focus in the acquisition process is beginning to transform. A new focus on 
SoS engineering takes systems engineering processes to the next level. In August 2008, 
The Director, Systems and Software Engineering Deputy Under Secretary of Defense 
(Acquisition and Technology) published Version 1 of the Systems Engineering Guide for 
Systems of Systems. Recognizing the growing interdependencies of systems to provide 
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warfighting capabilities, the guide serves as a first step to assist the systems engineering 
eommunity with adapting systems engineering processes to systems of systems 
engineering needs (OSD, 2008). 

The guide defines a SoS through reference to the 2008 DoD Defense Acquisition 
Guidebook (DAG), as “a set or arrangement of systems that results when independent 
and useful systems are integrated into a larger system that delivers unique capabilities” 
that remains unchanged in the 2010 update to the DAG. While this definition is not 
necessarily novel, the guide takes a step further by defining four types of SoS; Virtual, 
Collaborative, Acknowledged and Direeted, as depicted in Table 1. 


SoS T5q3e 

Description 

Virtual 

No central management authority or centrally agreed upon purpose for the SoS but 

displays large scale emergent behavior 

Collaborative 

Systems interact voluntarily to fill agreed upon central purpose 

Acknowledged 

Recognized objectives, designated manager and resources with individual systems 

retaining independent ownership, objectives, funding , development and sustainment 

Directed 

Integrated SoS built and centrally managed for a specific purpose with component 

systems retaining an ability to operate independently but with independent operational 

mode subordinated to SoS purpose 


Table 1. Types of SoS. 

Definitions of different types of System of Systems as diseussed in the 2008 Systems 
Engineering Guide for Systems of Systems (After; OSD, 2008, pp. 4-5). 


The Systems Engineering Guide for Systems of Systems recognizes that most 
systems today are linked through net-centric information sharing to form a Virtual SoS. 
Collaborative SoS result as greater interdependencies are reeognized and communities of 
interest are formed to work together for mutual benefit. Aeknowledged SoS are 
becoming more predominant in DoD aequisition, demonstrating eollaborative behavior, 
but with greater management authority and resources at the SoS level. Direeted SoS are 
less predominant. The Army’s Future Combat System (FCS) is offered as an example of 
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a Directed SoS, with individual systems development driven by the common objectives 
of the SoS (OSD, 2008). With budgetary and technology challenges that resulted in the 
cancellation of the PCS program in 2009, the Directed SoS may be the most challenging 
from an acquisition and management approach due to the inherent size and scope of a 
development effort at the Directed SoS level. 

In the Marine Corps, the Combat Operations Center (COC) and the Common 
Aviation Command and Control System (CAC2S) are two programs of interest from a 
SoS perspective. The COC is a fielded program of record that provides the C2 facilities 
for the commander and staff of the four components of the MAGTF; Command Element 
(CE), Ground Combat Element (GCE), Air Combat Element (ACE), and Eogistics 
Combat Element (ECE). The COC consists of physical components (tentage, power 
generation and environmental controls). Information technology infrastructure (servers 
and data storage), network and communications management capabilities (network 
management tools and interfaces for organic communications assets), and tactical 
software applications and associated computer platforms (PM MAGTE C2, 2009). While 
some of the component systems are unique to the COC, the C2-enabling capabilities are 
largely derived from components that are incorporated within the COC SoS but retain 
independent ownership, objectives, funding and sustainment. The Advanced Eield 
Artillery Tactical Data System (AEATDS), Command Post of the Euture (CPOE), Joint 
Automated Deep Operations Coordination System (JADOCS) and Global Combat 
Support System (GCSS) are a few examples of systems that are part of the COC baseline 
configuration but are managed as independent programs. Eurther, the wide-band 
communications infrastructure necessary to interface and tie individual COC together is 
also comprised of separately managed independent programs. Based on that, the COC, as 
seen in Eigure 2, would most closely represent an Acknowledged SoS. 
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Figure 2. Combat Operations Center (COC) OV-1. 

The COC provides C2 facilities for the components of the MAGTF 
(From; Zapata, 2010a). 

The Marine Corps CAC2S program depicted in Figure 3 represents a more 
ambitious approach. The initial effort focused on a complete replacement for C2 
equipment of the Marine Air Command and Control System (MACCS). With an intent 
to replace legacy single-mission systems with an integrated hardware and software 
solution for more effective command, control and coordination of air operations, this 
effort was most closely aligned with a Directed SoS development approach. However, 
much like FCS, CAC2S faced program management and technical challenges that 
required the program to restructure in 2009. With that restructure, CAC2S is now linked 
to the COC program, leveraging the C2 capabilities provided by the COC to meet some 
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of the C2 needs of the MACCS mission. Subsequently, with that dependeney on another 
SoS, CAC2S would now also be considered an Acknowledged SoS. 



Figure 3. Common Aviation Command and Control System (CAC2S) OV-1. 
The CAC2S represents an incremental SoS development effort that will replace the 
current Marine Air Command and Control System. In Increment I, Phase I, the 
functionality of the Direct Air Support Center (DASC) is replaced by CAC2S (From; 

Zapata, 2010b). 


In the Draft 7 January 2010 Mission Area “Systems of Systems” Systems 
Engineering Guidebook, the concept of mission capability is reinforced; 

In the future, global operations will be conducted by distributed, integrated 
and interoperable forces. This future warfare is about capability delivered 
by ‘System of Systems’ operating as a single system. System of Systems 
(SoS) is defined in this document as a force package of interoperable 
platforms and nodes acting as a single system to achieve a mission 
capability, i.e. a mission level SoS. Typical characteristics include a high 
degree of collaboration and coordination, flexible addition or removal of 
component systems, and a net-centric architecture. The capabilities 
provided by each constituent system operating within the SoS are framed 
by the integrated force package architecture. (Chief Systems Engineer, 

2010, p. 1) 
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For the Marine Corps, the MAGTF serves as the foundational strueture that 
provides a unique combined arms air-ground warfighting capability. As depicted in 
Figure 4, each MAGTF is comprised of four core elements: the Command Element (CE) 
that provides for the overarching command of the MAGTE; the Ground Combat Element 
(GCE) with infantry, artillery and armor; the Aviation Combat Element (ACE) with fixed 
wing, rotary wing and aviation command and control; and the Logistics Combat Element 
(LCE) with supply, combat engineers, medical and other combat services capabilities. 
The basic premise is that this structure provides a mission capability that is greater than 
the sum of the individual components: the very essence of a SoS. 



GCE ACE LCE 

I'lxcd Wing, Combat KngInccT, 

Inlantry,Artillery Rotary Wing and Medical nnd 

aodAnnor AruilioaC2 . other Services 


Eigure 4. MAGTF Components. 

Graphical representation of MAGTF hierarchical structure. 

While every MAGTF retains this basic structure, the size and individual 
component structure of the MAGTF is mission dependent and can range from largest— 
Marine Expeditionary Eorce (MEF) level (approximately 45,000 personnel)—to the 
smallest. Marine Expeditionary Unit (MEU) level (approximately 2,500 personnel). 
Both COC and CAC2S support this dynamic structure through modular and scalable 
design. The COC in particular is fielded in five different capability sets that are equated 
to an echelon of command; MEE COC (Capability Set I), Division COC (Capability Set 
II), Regimental COC (Capability Set III), Battalion COC (Capability Set IV) and 
Company COC (Capability Set V). A MEE level MAGTE with a MEE headquarters. 

Marine Division, Marine Aircraft Wing and Logistics Support Group would then consist 
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of multiple COCs in all five Capability Set variants. The CAC2S is fielded in a similar 
scalable manner with four different variations to accommodate C2 requirements for 
specific MACCS functions at Wing, Squadron, Echelon and Sub-Agency levels. 

From a warfighting capability perspective, we can view the MAGTF and all its 
people, processes and technology as an independent SoS, fully capable of executing 
warfighting mission capabilities in support of defined operational objectives. The 
MAGTF SoS can also be considered an element of larger Joint SoS when employed as 
part of a Joint Task Force. 

MAGTF C4 can be considered a SoS (within context of the greater MAGTF SoS) 
that is specifically architected to deliver a C2 capability in support of the C2 warfighting 
function. MAGTF C4 as a SoS is comprised of people (C4 operators and maintainers), 
processes (guided by doctrine and written procedures) and technology (communications 
systems, networking systems and C2 applications) that are brought together through an 
integrated MAGTF architecture that enable unique C2 functions and capabilities that 
cannot be achieved individually. Both COC and CAC2S (or currently MACCS) SoS, are 
components of the larger MAGTF C4 SoS, and together comprise a significant part of the 
MAGTF C4 SoS, with the intent of providing C2 capabilities at all echelons of command 
down through the Company level. 

This is precisely the concept that is presented in the Marine Corps’ MAGTF C2 
vision. In 2008, a MAGTF C2 Transition Task Force was established to provide 
governance and oversight to the process of developing and fielding systems that support 
C2 capability. With the intent of harmonizing the development and fielding of C2 
programs of record like CAC2S, COC and many others to deliver an end-to-end 
integrated C2 solution, this effort provides the Marine Corps with a means for 
transitioning from an Acknowledged to a more Directed SoS. Cloninger (2009) describes 
the mid-term goal of MAGTF C2 as an integrated SoS: 

MAGTF C2 will become an integrated C2 solution that will migrate the 
current multiplicity of stove-piped, disparate systems into an integrated 
system-of-systems that will support deployed aspects of Marine Corps C2 
requirements from pre-deployment planning to execution and 
redeployment via multi-functional C2 nodes, (p. 103) 
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From this perspective, MAGTF C2 is seen as the end state of the transition of 
MAGTF C4 SoS from an Acknowledged to a MAGTF C2 Directed SoS. However, until 
that end-state is achieved, the MAGTF C4 SoS (as depicted in Figure 5) then is 
considered an Acknowledged SoS, comprised of both SoS and individual systems 
centrally employed and managed through the MAGTF (as a hierarchical organization) 
with a recognized objective to provide a C2 capability that maximizes combat capability 
through unity of effort. 
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Ai my Component 
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Figure 5. MAGTF C4 SoS. 

The MAGTF C4 SoS provides a C2 capability to the MAGTF SoS that in concert with 
the Army and Air Force Component SoS make up the Joint Task Force SoS. 


C. JOINT LEVEL SOS TEST AND EVALUATION APPROACH 

While individual systems development with associated developmental test and 
operational test and evaluation (OT&E) is still the predominant focus in the DoD 
acquisition community, the reality of net-centric SoS warfare and the relevance of SoS 
performance to the ability to achieve mission capabilities have not gone unaddressed. 
Significant efforts, both individual and collaborative, to test, measure and quantify the 
performance of complex SoS have been, and continue to be, a major focus of both 
Component and Joint DoD testing organizations. At the Joint level, this effort has 
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predominantly fallen under the purview of the Under Seeretary of Defense / Aequisition 
Teehnology & Logisties (AT&L) and Direetor Operational Test & Evaluation (DOT&E). 

AT&E established the Joint Mission Environment Test Capability (JMETC) in 
2006 to provide the infrastrueture and eommon environment for distributed Live, Virtual 
and Construetive (LVC) testing in the Joint DoD eommunity. JMETC enables 
distributed faeilities to link together through the existing DoD Seeure Defense Researeh 
and Engineering Network (SDREN) to provide a Joint environment in support of 
“Developmental testing, Operational Testing, Interoperability Certifieation, Net-Ready 
Key Performanee Parameters (KPP) eomplianee testing, and Joint Mission Capability 
Portfolio Testing” (Loekhart & Eerguson, 2008, p. 161). JMETC offers a eorporate 
approaeh to Joint testing, providing readily available eonneetivity over existing DoD 
networks, a eommon middleware for eonneeting distributed faeilities, data exehange 
standards, data management, reuse repository and a suite of test tools for test planning, 
execution and analysis as described by (Lockhart & Eerguson, 2008): 

• Connectivity . The Secure Defense Research and Engineering Network 
(SDREN) provides connectivity between test facilities. Training facilities can 
also be linked through the Joint Training and Experimentation Network 
(JTEN). 

• Middleware . The Test and Training Enabling Architecture (TENA) serves as 
common data exchange environment used by distributed facilities to send and 
receive data. TENA provides gateways to connect to other data exchange 
protocols to include Distributed Interactive Simulation (DIS) and High-Level 
Architecture (HEA). 

• Data Exchange Standards . Through TENA Object Models, a common 
language is provided for data exchange between systems in the distributed 
architecture. 

• Data Management . JMETC provides a capability to archive test data from 
multiple facilities for analyses and evaluation of test results. 
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• Reuse Repository . A Web portal provides a means to share objeet models, 
middleware, software tools and doeumentation aeross the user eommunity. 

• Test Tools . A eommon suite of test tools is provided for planning, 
configuration, monitoring, control and analysis. The Interoperability Test and 
Evaluation Capability (InterTEC) is a secondary initiative under AT&E that 
started as a joint effort between the Joint Interoperability Test Command 
(JITC) and Naval Air Warfare Center (NAVAIR) and is now an integral part 
of JMETC and synonymous with the common test tool suite. Erom the JITC 
Web site, the InterTEC tool suite also provides the synthetic battlespace 
environment that shapes the test scenario through many-on-few scenario 
simulations and a simulation/emulation gateway to stimulate system sensors 
in the systems under test (“JITC,” 2009). 

In an effort complimentary to JMETC’s Joint test infrastructure and common 
EVC test environment, the DOT&E Joint Test (JTEM) effort was chartered in 2006 for 
three years, to develop, test and evaluate methods and processes for conducting EVC 
Joint test events. The principal product that resulted from this effort was the JTEM 
Capability Test Methodology (CTM) that provides a framework for a capabilities-based 
approach for testing in a Joint mission environment. CTM version 2.0 describes a six- 
step methodology, consisting of 14 JTEM processes, as depicted in Eigure 6. 
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Figure 6. ITEM CTMv2.0. 

The six-step ITEM Capability Test Methodology offers a struetured approaeh for 
conducting an SoS assessment (Erom; Eiebrandt & Dryer, 2009, p. 3). 


The six steps of the CTM provide a basic framework for any SoS evaluation. 
Outlined in JTEM’s Action Officer’s Handbook for Testing in a Joint Environment 
(Eorenzo, 2009), the CTM steps include: 

• Develop T&E Strategy . This step was added in CTMv2.0 to address a critical 
first step to define the overall SoS evaluation strategy. The key aspects of this 
step are to define the SoS in terms of capability and define the operational 
context for SoS employment. 

• Characterize Test. Once the SoS is defined, determining what aspects or 
attributes of the SoS can and should be measured to assess the SoS 
performance is a second key step. Without defined SoS measures of 
performance (typical of a non-Directed SoS), this can present the most 
challenging aspect of the methodology. 
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• Plan Test. The test plan that results from this step doeuments the overall test 
strategy, test design, and data collection methodology that guides test 
execution. 

• Implement LVC Distributed Environment. JMETC facilitates implementation 
of the LVC distributed environment. 

• Manage Test Execution. The Inter TEC tool suite facilitates test management 
and execution. 

• Evaluate Capability . The Inter TEC tool suite facilitates data collection across 
the LVC distributed environment, however the significance of the evaluation 
is largely dependent on how well the performance measures were defined in 
the second step. 

In 2008, ITEM demonstrated CTM version 2.0 in the ECS Joint Battlespace 
Dynamic Deconfiiction (JBD2) test event. The event provided an opportunity for both 
ITEM and JMETC to mature their products and offered ECS an opportunity to conduct a 
complex SoS test event in support of an acquisition Milestone C decision. Hutchison, 
Lorenzo and Bryan (2009) describe the complexity of the event: 

To achieve these goals, JBD2 established a complex joint mission 
environment composed of 16 test sites and more than 40 unique live, 
virtual, and constructive systems connected across four time zones. These 
test sites represented all four Services and the U.S. Joint Eorces 
Command. Of these 16 JBD2 sites, 10 were reused from two previous test 
venues that were a part of a series of events culminating in JBD2. Seven 
Service and joint initiatives were included as part of the test architecture. 

JBD2 truly provided joint context and stressed the boundaries of a live, 
virtual, and constructive joint mission environment, (p. 32) 

Eor the ECS JBC2 event, the SoS under test provided a C2 capability that was 
focused on two factors: 

• Battlespace Management Capability . Test to determine whether there was a 
difference between current and future SoS implementation during execution 
of mission tasks during the mission-based test scenario. 
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• Timeliness of C2 processes . Test to determine difference between current 
procedural-based processes versus future expedited-based processes (LeSueru, 
Millich & Stokes, 2009). 

Of significance is that this event demonstrated test and evaluation of both material 
(SoS physical configuration with respect to battlespace management capability) and non¬ 
material (C2 processes) aspects of the SoS. Also relevant is the use of a complex, but 
operationally realistic mission scenario as depicted in Figure 7, which provided the basis 
for stressing the SoS under test. 



Figure 7. JBD2 Operational View. 

Operationally realistic mission thread from Joint Battlespace Dynamic Deconfliction Test 
Event (From; LeSueru, Millich & Stokes, 2009, p. 3). 


D. COMPONENT (MARINE CORPS) SOS TEST & EVALUATION 
APPROACH 

At the Component level, the Marine Corps has taken a similar approach in both 
past (FEDOS) and current (MC3) SoS assessment efforts. Both efforts employed a 
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structured, process driven methodology and both were conducted in a test environment 
that attempted to replicate a MAGTF C4 SoS architecture. Although developed prior to 
and independent of ITEM, initial meeting with the ITEM program manager in January 
2007 revealed significant parallels in approach and methodology: 

• SoS capability-based assessment approach 

• SoS architecture defined and rooted in individual systems’ JCIDS artifacts 
(Operational Views and System Views) 

• Operational scenario and mission tasks provide basis for evaluating the 
performance of the SoS 

• Employed distributed test architecture with centralized test control 

The significant similarities in methodology for SoS testing at the Joint level to the 
methodology developed by MCTSSA for SoS testing at the Component level, indicated 
potential for mutual benefit through continued collaboration. In October 2007, MCTSSA 
joined the JMETC community and began participation in JMETC test events as the 
Marine Corp component (MAGTE node) in the Joint test architecture. This served as an 
opportunity for the Marine Corps to further mature its SoS assessment methodology and 
leverage the capabilities provided by the JMETC InterTEC tool suite. The collaboration 
also highlighted a number of shared challenges: specifically the development of 
meaningful test threads, selection of SoS performance metrics and development of a 
comprehensive and consistent manner for characterizing the results of the SoS 
performance assessment. 

E. GAP ANALYSIS (ASSESSMENT ISSUES AND DEFICIENCIES) 

The three shared challenges associated with SoS assessment (test threads, 
performance metrics and assessment reporting) are closely interrelated and improvements 
in one area could potentially yield improvement in the other areas. Erom a Component 
(MAGTE C4 SoS) perspective, a more detailed discussion of these challenges follows. 
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1. Operational Test Thread Concept 

A central feature of the Marine Corps’ C4 SoS testing methodology is the use of 
MAGTF-based capability mission tasks executed in an operational scenario. Translating 
the mission tasks to an operationally relevant test thread has been a challenge for a 
number of reasons: 

• Eaeh MAGTF is unique . While each MAGTF consists of the same eore 
elements (CE, GCE, ACE and ECE), the size and speeific component make 
up varies as dictated by Mission requirements. 

• Eaeh MAGTF C4 arehitecture is unique . Implementation of the C4 
components in the MAGTF architecture to provide the C2 capability, though 
guided by doetrine and best practices, is left to the diseretion of the MAGTF 
eommander. 

• Execution of mission tasks can vary . Mission tasks are well defined but how 
individual C4 components are used in eoneert with the execution of the tasks 
is often not defined. 

All these factors come into play during the development of a test thread that 
replicates execution of a mission task in the test environment. Assumptions to 
aceommodate all these faetors are required as the test agency translates the mission tasks 
into a form that ean be exeeuted in the test seenario: mission task translated to test thread. 
Once developed, the abstracted test thread must then be validated to ensure it is still 
operationally relevant. During the C4 SoS assessment, significant resources (personnel 
and equipment) may be required to exeeute the test thread depending on degree of human 
to machine interaction necessary to eomplete the mission task. 

To provide a true representation of the MAGTE C4 SoS in an operational eontext, 
a variety of mission tasks running in parallel and at various stages of completion are 
required. Multiple mission tasks, executed in eoneert with a defined MAGTE battle 
rhythm, would provide a meaningful eontext for assessing the performanee of the C4 
SoS. However, the effort and resourees required to develop, validate and execute 

sufficient test threads to represent this is considerable and possibly infeasible without the 
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aid of modeling and simulation. Further, even with the MAGTF C4 SoS well defined 
and stressed through exeeution of realistie mission tasks, determining what to measure 
from a C4 SoS performanee perspeetive, is still an issue without doeumented mission- 
task performanee metries to test against. 

2. SoS Performance Metrics 

In response to the laek of defined mission task performanee metries, the Marine 
Corps’ 2007 MC3 MAGTF C4 SoS effort developed a unique metrics model in 
collaboration with stakeholders within the Marine Corps Combat Development 
Command (MCCDC), SIAT, MCTSSA, Marine Corps Operational Test and Evaluation 
Agency (MCOTEA) and Space and Naval Warfare (SPAWAR) Systems Center, Atlantic. 
The metrics model was intended to focus on characterizing the performance of the SoS in 
terms of mission-based test thread execution, vice performance of individual C4 systems 
and was comprised of five areas of measurement: 

• Operator Complexity . Measure of level of difficulty to execute a test thread 
from a user’s perspective. The measure was based on the total number of 
operator steps required at each system or node in the SoS during execution of 
the test thread. 

• System Timeliness . Measure of total system time to execute the test thread. 
This measure was based on total digital transmission time and system 
processing time at each system or node in the SoS during execution of the test 
thread. 

• System Accuracy . Measure of application layer accuracy as indicated by 
percent completion of digital messages between each system and node in the 
SoS during execution of the test thread. 

• System Reliability . Measure of number of failures at each system or node in 
the SoS during execution of the test thread. 

• Anomalous Behavior . Not a measure but a means to capture emergent or 
unexpected behavior within the SoS during execution of the test thread. 
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Although successfully demonstrated in a final test event, the value of the model 
was limited. While the intent was to establish a metrics model that focused on SoS level 
mission thread execution, the resulting SoS performance assessment was extrapolated 
from the performance of the individual systems in terms of the metrics as they eompleted 
tasks during execution of the test thread. Based on the author’s personal observations 
and involvement with the 2007 test event, this shifted foeus away from the SoS 
performanee to a study of performanee of individual systems operating within a SoS. 

The MAGTF C2 working group provided another metrics model. In the MAGTF 
C2 Test and Evaluation Master Plan for 2010, five Crhieal Operational Issues (COIs) are 
defined that guide the seleetion of Key Performanee Parameters (KPPs) with assoeiated 
Measures of Performanee (MOPs): 

1. To what level does MAGTF C2 enable Blue Foree Tracking of 
friendly assets? 

2. To what level does MAGTF C2 provide foree and unit 
eommanders a Common Taetieal Pieture / Common Operational 
Picture? 

3. To what level does MAGTF C2 provide adequate Situational 
Awareness to unit eommanders and their forees? 

4. To what level does the MAGTF C2 provide planning and 
eollaborative funetionality in the net-eentrie, serviee-oriented 
environment to distributed COCs? 

5. To what level does MAGTF C2 provide the eapabilhy to 
oommunieate while on the move? (MAGTF C2, 2008, pp. 21-23) 

Because the MAGTF C2 effort is primarily an SoS engineering oversight effort 
for a number of existing programs of reeord (CAC2S and COC serving as the primary 
programs of reeord), the KPPS and MOPs assoeiated with COIs’ 1,2,3 and 5 were 
extracted from the existing programs of reeord Capability Development Doeuments 
(MAGTF C2, 2008). COI 4 required ereation of new MOPs aligned with the Net-Ready 
KPP defined in CJCSI 6212.OlE. The Joint Interoperability Test Command (JITC) is 
responsible for eonduoting interoperability evaluations of programs of reeord to certify 
eompliance with the five elements of the NR-KPP; 
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1. Compliant solution architecture: teehnieal exehange of information 
and end-to-end use of that exehange. 


2. Complianee with net-eentrie data and serviees strategies; 
evaluation of net-centric data and services to determine if net- 
ready. 

3. Complianee with applieable GIG Teehnieal Guidanee (GTG). 

4. Complianee with DOD Information Assuranee (lA) requirements 

5. Complianee with supportability requirements to inelude spectrum 
utilization and information bandwidth requirements, Seleetive 
Availability Anti-Spoofing Module (SAASM) and the Joint 
Taetieal Radio System (JTRS), as applieable. (CJCSI 6212.OlE, 

2008, p. A-2) 

For the MAGTF C2 effort, the NR-KPP eertifieation requirements for the 
individual programs of record within the SoS (ineluding COC and CAC2S) were adapted 
to address NR-KPP complianee from a SoS perspeetive. This methodology differs 
signifieantly from the approaeh taken in MC3. While MC3 attempts to charaeterize the 
performanee of the SoS in terms of mission thread exeeution (warfighting eapability), the 
MAGTF C2 approaeh is more oriented towards assessing the C2 eapability provided by 
the SoS. 


3. Reporting SoS Performance Results 

To characterize and quantify the performanee of the C4 SoS, MCTSSA’s MC3 
effort applied a risk-based reporting methodology. During the assessment event, 
mission-based test threads were executed through the C4 SoS and data was oolleeted 
based on the established areas of measurement defined for the event (operator 
eomplexity, system timeliness, system aecuracy, system reliability and anomalous 
behavior). The data was summarized and evaluated by a panel of stakeholders and 
experts to derive an overall risk score assoeiated with exeeuting the test thread in the C4 
SoS. The overall results of the assessment effort were reported in a standard 5x5 risk 
ehart in a manner similar to that depleted in Figure 8. 
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MAGTF C4 SoS Mission Thread Execution Performance 
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Figure 8. MC3 Risk-based SoS Performance Assessment Reporting. 
MCTSSA’s MC3 SoS Assessment risk chart depicts risk associated with executing a 
given test thread within a defined SoS architecture. 


Without defined performance criteria associated with execution of the test thread, 
the interpretation of the data to determine a risk measure was subjective and served only 
as a characterization of risk associated with execution of the test thread. Performance 
data obtained from the individual systems during execution of the test thread was used to 
develop a measure of risk associated with the success of executing a test thread in the 
SoS. The overall risk associated with executing Test Thread D (Figure 8) in the C4 SoS, 
for example, is moderate with a low likelihood of failure, but catastrophic consequence to 
the mission when it fails. Through this methodology an overall assessment of the 
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performance of the SoS could be obtained by taking an average or weighted average of 
the total test thread risk scores to provide an overall C4 SoS risk score for use as the SoS 
performance metric. 

The intent was that this method would provide a level of abstraction that would be 
more acceptable to the key stakeholders (Program Offices with ownership of individual 
programs within the SoS), than a typical pass/fail grading criteria normally associated 
with formal test activities. While this reporting methodology does serve that purpose and 
does provide a means to characterize the overall SoS performance in terms of mission 
task execution, other key aspects of the C4 SoS; net-centric C4 SoS effectiveness and 
suitability attributes like reliability and timeliness measures used to obtain the risk 
measure for thread execution in the SoS are obscured. 

4. DoD and Joint Approaches to the Challenges 

In the 2008 Systems Engineering Guide for Systems of Systems, a more 
operational user-centric approach to addressing the challenge of SoS is recommended: 

Because acknowledged SoS typically comprise existing (often fielded) 
systems (e.g., AOC, SIAP, MILSATCOM), data from operations is an 
important source of understanding the state of the SoS. Because the SoS 
will likely evolve based on incremental changes in individual systems, it is 
important to have a set of user-oriented metrics that can be applied in 
different settings over time. The SoS systems engineer uses data from 
these settings to analyze SoS performance and behavior; hence, the 
metrics should include measures that use data from operations. These SoS 
metrics should also be traceable to the capability objectives established for 
the SoS, and there may even be a need to rank the metrics by importance. 

These metrics should not change as the capability of the SoS matures 
unless the capability objectives themselves change. They must remain 
applicable as the SoS matures to assess whether the changes made are 
actually translating into better user support. When captured in an 
operational environment, metrics allow an independent view to assess SoS 
performance from the user’s perspectives, and allow assessment of the 
impacts of external factors on capability objectives. These operational 
user-based performance assessments do not substitute for the technical 
reviews and assessments performed during the process of upgrading the 
systems in the SoS. (OSD, 2008, p. 44) 
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This guidance implies greater relianee on users to define performanee metries of 
the SoS during employment in an operational eontext. As an approaeh reeommended for 
an Aeknowledged SoS, this would be very applieable to a MAGTF C4 SoS, however the 
measured attributes of the SoS (whieh are not defined by this guidanee) would need to be 
earefully seleeted to aeeommodate the inherent variations in MAGTF C4 SoS 
implementation. User provided data on network loading and battle-rhythm indueed 
variation in network-data traffie patterns would be direetly applieable to MAGTF C4 SoS 
testing; providing operationally realistie base-line data that ean be used to stress the SoS 
during assessment. 

When ITEM reaehed the end of its three-year eharter in 2009, the effort to 
eontinue work towards SoS test and evaluation eontinued at the Joint level through a 
DOT&E Speeial Projeet: Joint Test and Evaluation Methodology - Transition (JTEM-T). 
Chartered through 2011, JTEM-T eontinues to refine the JTEM CTM eoneept with 
speeifie foeus on deeomposing “...Joint Mission Threads (JMT) into mission- and 
tasked-based testable measures” (Walters, 2010, p. 3). The JTEM-T effort established a 
metries working group with the goal of developing by the end of Eiseal Year 2010; 

• A repeatable proeess for deeomposing JMTs into testable measures 

• Aetual testable measures for seleeted JMTs 

• A report to DOT&E on reeommended ehanges to the Joint 
Capabilities Integration and Development Systems to faeilitate 
testing in a joint environment 

• A report through DOT&E to the OUSD/AT&E TRMC 
reeommending the tools and instrumentation needed to support the 
use of JMT testable measures. (Walters, 2010, p. 4) 

The signifieanee of this is the promise of a database of mission-based test threads and 
assoeiated metries for warfighting eapability-based SoS assessment at the Joint and 
Component level. These test threads may also be adapted for MAGTE C4 SoS 
assessment, providing stimulus and another measure of performanee during C4 SoS 
assessment. 
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At the Joint level, progress has been demonstrated towards maturation of the 
infrastructure (through JMETC), methodology (through ITEM) and initial mission- and 
tasked-based testable measures (through JTEM-T) necessary for Joint SoS test and 
evaluation. While these products may be appropriate for adaptation by the Marine Corps 
for a MAGTF SoS assessment, they do not provide a complete solution for assessing the 
MAGTE C4 SoS. At the JTE or MAGTE level, the mission task itself serves as focus of 
assessment as a measure of the overall warfighting capability (how well, how quickly, 
how efficiently the SoS prosecutes the mission). The MAGTF C4 SoS contributes 
towards the overall warfighting capability by providing a C2 capability and, from that 
perspective, metrics associated with assessing the SoS in terms of providing the C2 
capability (in line with the MAGTF C2 SoS assessment approach) would seem the most 
appropriate. However there is also value in assessing the performance in terms of 
mission thread execution and net-centric C4 SoS effectiveness and suitability (in line 
with the MC3 effort) (Figure 9). 



A complete MAGTF C4 SoS assessment should include metrics related to net-centric 
effectiveness and suitability, mission thread execution and C2 services. 
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Documenting and reporting the results of the SoS performanee assessment in a 
eomprehensive, eonsistent and useful manner eontinues to be a ehallenge. This ehallenge 
is due in part the fluidity of the SoS assessment approaeh (defining test threads and 
metries) but is also due to the eomplex nature and broad seope assoeiated with 
eharaeterizing the behavior of a large seale SoS. Charaeterizing a SoS in terms of risk 
that may be deereased or inereased through a ehange in SoS baseline eonfiguration may 
have some value. Other approaehes, sueh as eharaeterizing the SoS in terms of 
interoperability, may also have value and are the subjeet of a number of independent 
efforts in DoD and the eommereial seetor. 

F. SUMMARY 

As the definition of a SoS within a DoD operational eontext has evolved and 
matured, so have the engineering approaehes to developing and assessing a SoS. At the 
Joint SoS level, the warfighting eapability serves as the foeus of assessment, enabled by 
the JMETC distributed SoS testing infrastructure, InterTEC test tool suite, ITEM proeess 
and proeedures and JTEM-T Joint Mission Threads (JMTs). This Joint SoS assessment 
methodology ean be leveraged for Component SoS assessment (MAGTE as the Marine 
Corps Component) by adapting appropriate mission threads to the eomponent warfighting 
eapabilities. 

Eor the MAGTE C4 SoS, the C2 eapability that enables the C2 warfighting 
funetion provides the foeus for SoS assessment. Exeeuting mission-based test threads 
within a MAGTE C4 SoS that is stressed through an operationally realistie variable 
network load and data traffie patterns, provides an appropriate foundation for an 
assessment. Determining the speeifie attributes of the SoS to measure eontinues to be an 
evolving proeess. Eor the Marine Corps’ MAGTE C4 SoS, a blend of a number of 
approaehes may provide a more holistie assessment of the SoS. With an assessment of 
mission thread exeeution (timeliness, eompletion pereentage), C2 serviees (situational 
awareness, position reporting, eollaboration, and other Defense Information Systems 
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Network (DISN) services) and net-centric C4 SoS effectiveness and suitability 
(bandwidth efficiency, network reliability), a more comprehensive measure of SoS 
performance can be obtained. 

As the techniques, processes and procedures for executing SoS assessments 
mature, there is an increasing need to convey the assessment results in a context that 
provides a more useful and meaningful characterization of the SoS performance. In the 
next sections a select sampling of commercial tools and techniques will be examined that 
may aid in this effort. 
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III. CURRENT ASSESSMENT TOOLS WITH POTENTIAL 
APPLICABILITY TO MARINE CORPS C4 SOS PERFORMANCE 

ASSESSMENT 


A. INTRODUCTION 

The Marine Corps C4 SoS performance assessment effort contributes to ensuring 
that the fielded C4 SoS provides an end-to-end integrated C2 solution for the Marine 
Corps Operating Forces. Optimally this assessment would address all aspects of the SoS: 
people (C4 operators and maintainers), processes (guided by doctrine and written 
procedures) and technology (communications systems, networking systems and C2 
applications). However, in that respect, even the assessment of the smallest MAGTF 
(MEU), would require considerable resources to include as many as 2500 operators. In 
practice then, the SoS assessment out of practical necessity must be conducted through a 
modeled abstraction of a MAGTF C4 SoS. Typically, the representative MAGTF C4 
SoS designed for the assessment is a sampling of equipment and software applications 
sufficient for executing select mission-based test threads and providing sufficient 
interactions to invoke the SoS behavior necessary to provide a basis for performance 
measurement. Furthermore, the assessment focuses primarily on the technology aspects 
of the SoS (the material solution for C2) to minimize the need for end-user operators and 
maintainers and lessen impact to the Marine Corps Operating Forces. Other key aspects 
of the SoS (both people and processes) serve as acknowledged constraints to the conduct 
of the SoS assessment and require assumptions regarding operator training level and 
fidelity of the processes associated with the SoS. While these aspects are integral to the 
overall SoS performance, a more narrow focus on the technical aspects of the MAGTF 
C4 SoS provides a more granular evaluation of this aspect than typically afforded by a 
formal MCOTEA Operational Test and Evaluation capability-based test of effectiveness 
and suitability. 

Even with this focused approach and corresponding reduction in scope and 
complexity, characterizing the performance of the SoS in terms of the criteria associated 
with the technology is a challenge in and of itself MCTSSA’s MC3 assessment effort 
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attempts to do this through a composite of various performance metrics to produce an 
overall risk-based assessment characterization of the SoS. This approach, while relating 
SoS performance to execution of a mission-based test thread and providing some level of 
traceability of the thread execution to the contributions of individual systems within the 
SoS, has only been demonstrated in the Marine Corps MC3 effort with relatively small 
numbers of systems (5-10) in the SoS test framework. To assess a SoS during execution 
of more complex mission-based test threads or simultaneous execution of multiple test 
threads that involve significantly more systems as well as invoke more complex network 
behaviors, the need for assessment tools becomes evident. In addition to helping with 
management of greater quantities of data, assessment tools can help with standardizing 
the methodology for reporting, weighting and summing the SoS metric data for an overall 
SoS performance assessment. A sampling of some assessment tools that attempt to do 
just that is examined below. 

B. ASSESSMENT TOOLS 

A theme common to a number of the tools proposed for SoS assessment is the 
idea of using interoperability as an overall SoS performance measure. However, much 
like the definition of SoS, interoperability must also be defined in a relevant context. The 
DoD Dictionary of Military and Associated Terms defines interoperability as: 

1. The ability to operate in synergy in the execution of assigned tasks. 

2. The condition achieved among communications-electronics 
systems or items of communications-electronics equipment when 
information or services can be exchanged directly and 
satisfactorily between them and/or their users. (Joint Chiefs of 
Staff, 2009) 

The first definition aligns nicely with the basic definition and concept of a SoS, 
the second with the focus on the technology aspects of a SoS during an assessment event. 
Both definitions would then seem to imply that interoperability could serve as a focus for 
assessing overall SoS performance. 

In 2004, the DoD contracted Carnegie Mellon Software Engineering Institute to 
examine the concept of interoperability and define a SoS interoperability (SCSI) model. 
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The model described in their final report incorporates aspects of technical 
interoperability, operational interoperability and programmatic interoperability. The 
SOSI model draws on technical and operational interoperability concepts introduced by 
the Levels of Information System Interoperability (LISI) model, NATO C3 Technical 
Architecture (NC3TA) Reference Model for Interoperability, the Organizational 
Interoperability Maturity Model (OIM) and Levels of Conceptual Interoperability Model 
(LCIM). The idea of programmatic interoperability is introduced as another dimension 
of interoperability by the SOSI model to recognize the influence that acquisition activities 
have on achieving the aspects of interoperability (Morris, Levine, Meyers, Place & 
Palakosh, 2004). Different levels of sophistication of SoS interoperability are central to 
all the interoperability models and closely parallel the concept of different types of SoS 
(Virtual, Collaborative, Acknowledged and Directed) defined in Systems Engineering 
Guide for Systems of Systems (OSD, 2008). The technical and operational aspects of 
interoperability also closely parallel aspects of how we view a SoS from an assessment 
perspective. This further strengthen the argument for using a measure of interoperability 
as a means of assessing overall SoS performance. 

1. i-Score 

Developed by Major Thomas Ford, Dr. John Colombi, Dr Scott Graham and Dr 
David Jacques from the Air Force Institute of Technology (AFIT), the Interoperability 
Score (i-Score) model is intended to offer a more quantitative measure of assessing the 
interoperability of a SoS than the qualitative measure (levels of interoperability 
sophistication) provided by models like LISI, OIM and SOSI (Ford, Colombi, Graham & 
Jacques, 2007). The model asserts a number of strengths to include: 

1. It is easily computed, 

2. It is based upon an operational thread, 

3. It makes use of existing architecture data, 

4. It can be used for scenarios where one or more type of 
interoperability is represented (i.e., information and organizational 
interoperability) 
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5. It defines the optimum interoperability for a given operational 
thread allowing a deeision maker to understand what the limits of 
his/her interoperability improvements are and what ean 
realistieally be done to improve interoperability for an operational 
interest and 

6. It provides a means of drilling down into a proeess to diseover 
where interoperability problems he. (p. 2) 

The model presents a mathematieal means for measuring the interoperability of a 
SoS obtained by analyzing the SoS components that contribute to the execution of a 
mission test thread. A mathematical score (i-Score measure) is calculated based on the 
values attributed to the interactions between components (value of -1,0 or 1 assigned 
depending on degree of data translation needed between components) and the number of 
times the component is used during execution of the mission thread. The i-Score 
measure is obtained through a six step methodology summarized by Ford, Colombi, 
Graham and Jacques (2006) as follows: 

Step 1 - Diagram the operational thread (e.g., time-critical-targeting) 
using an IDEFO, BPML, or UML activity diagram and define the ordered 
set T of systems supporting each activity in the thread. 

Step 2 - Create an interoperability matrix M = where n is the 

number of systems supporting the thread, C = [cy J„„ is a multiplicity 

matrix which describes the number of times a system is used in the thread, 
and = is a spin matrix where Sij, s{-1,0,1}, is a variable indicating 

no human or machine translation needed for a system pair (+1), machine 
translation required (0), or human translation required (-1). M can be 
augmented by multiplying additional matrices (layers) such as normalized 
bandwidth, probability of connection between system pairs, mission 
capable rate for systems, normalized cost, system reliability, etc. 


Step 3 - Calculate the i-Score 

^ij 

1=1 j=\ 


n n 

Step 4 - Calculate the optimum/-Score I om I where 

/ J \ tn^p, 

t=i y=i 
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CiiSu , , 

ij ij l5--=max{5'--} 


is the maximally upgraded interoperability 


matrix (i.e., upgrade all spins that can be upgraded in light of physical, 
fiscal and operational constraints). 


Step 5 - Calculate the Interoperability Gap I= 1^^, -1 . 


Step 6 - Perform interoperability analysis to 1) determine ways of closing 
the interoperability gap through spin upgrades or using common systems, 

2) determine average interoperability spin, 3) compare operational threads 
through a normalized i-Score, or 4) visualize the interoperability of a 
thread by graphing it on an Interoperability Terrain graph, (pp. 15-16) 

During a MCTSSA MC3 SoS assessment in 2008, in addition to the risk-based 
reporting methodology used to quantify the performance of the C4 SoS, a demonstration 
of the i-Score methodology was conducted using an Immediate Close Air Support (ICAS) 
mission-based test thread as the basis for the demonstration. The ICAS test thread 
(Figure 10) was simplified for this demonstration to ensure all activities during the 
execution of the mission thread were conducted in a serial manner since the i-Score 
methodology did not easily accommodate parallel processes (Sjoberg, 2008). 
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Figure 10. MC3 08 Modified ICAS Test Thread. 

Modified Immediate Close Air Support (ICAS) mission-based test thread used to 
demonstrate i-Score methodology during MCTSSA MC3 C4 SoS assessment in 2008 

(After; Sjoberg, 2008). 
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The systems supporting the thread aetivity were identified as the Forward Air 
Controller (FAC), denoted as System 1 (si); the Fire Support Coordination Center 
(FSCC), denoted as System 2 (S 2 ); the Direct Air Support Center, denoted as System 3 
(S 3 ) and the Aircraft (AC), denoted as System 4 (S 4 ). With the test thread defined, the six 
step i-Score methodology was applied as follows: 

Step 1. From the mission thread in Figure 10, the ordered set T of systems 
supporting the ICAS mission thread was determined as reflected in Equation 3.1: 

T'={Si,S3,S2,S3,S4,Si,S4,Si S4,S3} . (3.1) 

Step 2. The interoperability matrix (4x4 matrix: n=4 since we have 4 systems 
supporting the thread) is determined by Equation 3.2 (Eord et ah, 2006): 

M = , (3.2) 

L y y Anxn 

where, 

C = Multiplicity Matrix, 

S = Spin Matrix. 

The multiplicity matrix is determined by taking the elements of T two at a time in 
a forward direction (direction of mission thread execution) to determine set A (Eord et ah, 
2006). The intent is that this methodology accommodates both direct interaction as data 
is interchanged between components and indirect interaction that accommodates the 
upstream influences as the data progresses through the mission thread. Eor the ICAS 
thread then we determine set A in Equation 3.3 as follows: 

A = {(Si,S 3 ), (Si,S 2 ), (Si,S 3 ), (Si,S4), (Si,Si), (Si,S4), (Si,Si), (Si,S4), (Si,S3), 

(S3,S2), (S3,S3), (S3,S4), (S3,Si), (S3,S4), (S3,Si), (S3,S4), (S3,S3), (S2,S3), 

(§ 2 , 84 ), (S 2 ,Sl), (S 2 ,S 4 ), (S 2 ,Si), (S 2 ,S 4 ), (S 2 ,S 3 ), (S 3 ,S 4 ), (S 3 ,Si), (S 3 ,S 4 ), 

(53.51) , (S3,S4), (S3,S3), (S4,Si), (S4,S4), (S4,Si), (S4,S4), (S4,S3), (Si,S4), 

(51.51) , (Si,S4), (Si,S3), (S4,Si), (S4,S4), (S4,S3), (Si,S4), (Si,S3), (S4,S3)}. (3.3) 
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The pair (si,si) appears in set A three times whieh defines Ci,i, and pair (si,S 2 ) 
appears in this set one time whieh defines Ci ,2 and so on to result in the multiplicity 
matrix defined in Equation 3.4 (Ford et ah, 2006): 



3 15 6 

2 0 2 3 

4 13 6 

3 0 3 3 


(3.4) 


For the spin matrix, a pair-wise comparison and analysis was completed for each 
system and relative interoperability with each other. The results of the analysis are 
reflected in Table 2: 
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Spin 

element 

Spin 

Max 

Spin 

Rationale 

Sii 

1 

1 

All systems have perfect interoperability with themselves, which 
is to say Sij = Sii. 

S^2 

0 

1 

The FAC’s Target Location, Designation and Fland-Off System 
(TLDFIS) does not communicate directly to the FSCC’s 
Advanced Field Artillery Tactical Data System (AFATDS) or 
Intelligence Operations Workstation (lOW). Instead the FAC 
sends information directly to the DASC which forwards the 
information on to the FSCC. It is possible for the TLDFIS to 
communicate directly with the FSCC so this spin is upgradeable 
to 1. 

^13 

1 

1 

The FAC’s TLDFIS enables direct data exchange with the DASC. 

^14 

1 

1 

The FAC’s TLDFIS enables direct data exchange with the aircraft. 

■^21 

0 

1 

The FSCC must pass information through the DASC to 
communicate to the FAC. It is physically possible to establish 
digital communication from the FSCC to the FAC directly so this 
spin is upgradeable to 1. 

■^23 

1 

1 

The FSCC and DASC can conduct direct data exchange using 
lOWs and AFATDS. 

■^24 

0 

0 

The FSCC cannot conduct direct data exchange with the aircraft 
and must do so through the DASC. This is not upgradeable since 
there is no requirement for the FSCC to interoperate directly with 
the aircraft. 

■^31 

1 

1 

The DASC communicates directly with the FAC using lOW or 
AFATDS and TLDHS. 

^32 

1 

1 

The FSCC and DASC can interoperate using lOW and AFATDS. 

^34 

-1 

1 

The DASC only exchanges data with the aircraft using a human 
translator over voice channels. This spin is upgradeable since 
digital communication is possible between these two entities. 

■^41 

1 

1 

The aircraft can interoperate directly with the FAC’s TLDFIS. 

■^42 

0 

0 

The FSCC and aircraft cannot directly exchange data and must do 
so through the DASC. This is not upgradeable since there is no 
requirement for the FSCC to interoperate directly with the 
aircraft. 

5 43 

-1 

1 

The aircraft only provides data to the DASC using a human 
translator over voice channels. This spin is upgradeable since 
digital communication is possible between these two entities. 


Table 2. Spin Analysis. 

Analysis of the modified ICAS test thread results in assignment of Spin values for eaeh 
direet interaetion between eomponents in the test thread (From: Sjoberg, 2008). 
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From the spin analysis a spin matrix (5) and optimal spin matrix (Sopt) are 
determine in Equations 3.5 and 3.6 (Ford et ah, 2006): 



nxn 


10 11 
0 110 
111-1 
10-11 



1111 
1110 
1111 
10 11 


(3.5) 


(3.6) 


Now with the multiplieity matrix (C), spin matrix (5) and optimal spin matrix 
(Sopt) defined, the interoperability matrix (M) and optimal interoperability matrix (Mopt ) 
ean be determined as refieeted in Equations 3.7 and 3.8: 
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(3.8) 


Step 3. From M, an i-Score (/) as determined from Equation 3.9 (Ford et ah, 

2006): 
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(3.9) 


n n 

1=1 7=1 

is then calculated for the ICAS test thread as 7=21. 

Step 4. From Mopt , the optimum i-Score (lopt) as determined from Equation 3.10 
(Ford et ah, 2006): 


hp, 


(3.10) 


i=l 7=1 

is then calculated as I opt = 42. 

Step 5. The Interoperability Gap {Igap) as determined from Equation 3.11 (Ford et 
ah, 2006): 


^ gap ^ opt ^ 


(3.11) 


is then calculated with Igap= 21. 

Step 6. From the initial analysis, two interoperability factors were identified that 
could close the interoperability gap: the inability of the DASC to digitally communicate 
with the aircraft and the inability of the FSCC and FAC to directly exchange date. 
Addressing the first factor would close the Igap score by 18 points. Addressing the second 
factor was determined to have little operational relevancy (Sjoberg, 2008). 

To provide a higher level of fidelity to the model, an improved version was 
offered in 2008. The improved i-Score methodology replaces the discrete values of the 
spin matrix with continuous values that describe each system’s relative interoperability in 
what is then termed a resemblance matrix. The modification to the model is described by 
Ford, Colombi, Graham and Jacques (2008) as follows: 

The nature of each system is described as a set of system character states, 
which set is called a system instantiation. The resemblance of each 
instantiation pair is measured using a distance metric, and the resulting set 
of resemblance coefficients is given as a resemblance matrix. The i-Score 
interoperability measurement method is then upgraded by replacing the 
original spin matrix with the resemblance matrix. Because the coefficients 
in the resemblance matrix represent exact measures of similarity between 
systems, based upon system characters pertinent to interoperability, the 
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measure of interoperability obtained enjoys more fidelity and aeeuraey 
than that possible with the original spin-based method, (p. 2) 

As applied to the MAGTF C4 SoS assessment effort, i-Seore provides a 
methodology for determining and quantifying the relative interoperability of a mission- 
based test thread under the supposition that the optimal interoperability of any SoS is 
obtained with maximum direet digital eommunieations between eomponents (no human 
or maehine translation required). The maturity of the interfaee between systems or 
eomponents in the SoS serves as the basis for the SoS assessment then, with direet digital 
eommunieations representing the highest maturity and human translation representing the 
lowest maturity. From an operational thread perspeetive, there are often many ways to 
exeeute a mission or task with varying degrees of human interaetion (multiple 
eombinations of systems and proeedures available to aeeomplish the same mission). 
Subsequently, a test organization must seleet from a number of possible test threads that 
meet the mission requirement. In this respeet, the i-Seore methodology ean assist with 
defining the test threads that promise optimal interoperability (threads with least human 
translation required) and help narrow the seleetion for inelusion in a test event. 
Additionally, during exeeution of the test thread for an assessment event, the relative 
impaet of a deviation during thread exeeution (e.g., human translation required due to 
eomponent translation failure) eould be immediately quantified through a reealeulation of 
the i-Seore. Further analysis of the improved i-Seore methodology may yield further 
applieation with added value to a C4 SoS assessment effort. 

2. Interoperability Quotient 

The Interoperability Quotient (IQ) proeess was developed by Northrop Grumman 
Corporation to address eomplex SoS testing related to the Navy’s CVN-21 next 
generation Aireraft Carrier program. IQ was presented to the Joint test eommunity 
during the June 2007 JMETC User Group Conferenee in Dulles VA, as a more objeetive 
approaeh to performanee assessment during Joint SoS test events. Mueh like i-Seore, this 
methodology addresses the laek of a quantitative measure for assessing C4 SoS 
interoperability: noting that interoperability eomplianee with measures of performanee 
sueh as NR-KPP laek objeetive metries (Lawver, 2007). 
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The test proeess looks at interoperability from two perspeetives: first, from the 
perspective of data transfers between individual components of the SoS (external tests) 
and second, from the perspective of data processes within the individual components of 
the SoS (internal tests). This approach is referred to by Northrop Grumman as the 
“interoperability test stack” or “IQ Stack” to draw parallels to modular functional data 
models like the Open System Interconnection (OSI) 7-layer stack (Lawver, 2007). 

As depicted in Figure 11, the external tests look at data interaction between the 
systems within the SoS in terms of vulnerability, Web standards compliance, edge 
condition testing. Service Oriented Architecture (SOA) technologies, data transmission 
bandwidth, corrupted data and security. The internal tests look at attributes of the 
individual systems within the SoS in terms of Human Computer Interface (HCI) and 
Symbology, business rules for data fusion. Information Assurance (lA) and semantic 
interoperability (Lawver, 2007). 
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Figure 11. Northrop Grumman “IQ Stack”. 

The Northrop Grumman IQ Stack depicts aspects of both internal and external data 
interactions between components in a SoS (From; Lawver, 2007). 
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The IQ process is conducted within the context of an operational mission test 
thread that provides the basis for evaluation of the components (command centers, 
platforms, individual systems or applications) in the SoS. As the thread is executed, 
internal and external test artifacts are collected for the components and manually scored 
based on an established scoring criteria (numerical score associated with each possible 
outcome). As the numeric scores are determined, a color code (green-pass, yellow- 
caution, and red-fail) is also assigned. Both internal and external test scores are rolled up 
for an overall score and then those scores are rolled up to provide an overall component 
IQ score (maximum 200-point scale) and overall color code for the component (Lawver, 
2007). This roll up methodology as depicted in Figure 12, also allows for specific 
weighting of individual tests to accommodate specific design capabilities to help 
normalize the score to the 200-point scale (Lawver, 2007). Of note is that the validity of 
the roll-up methodology is dependent on the scoring methodology selected for the 
internal and external tests. If scoring is not based on a ratio scale, the summation and any 
statistical inferences will not be valid. 



Figure 12. IQ Computation Roll Up Methodology. 

IQ rollup methodology is depicted for the GCSS-M component in the CVN-21 SoS 

(From; Lawver, 2007). 
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To help better visualize and report the results of an interoperability assessment, 
Northrop Grumman also developed a companion Web portal tool called “Auto IQ” 
(Lawver, 2007) that provides a visual interpretation of the data and allows drill down to 
see how the individual scores are rolled up for the overall IQ component score. Depicted 
in Figure 13, the intent is to provide a means for analysts to drill down into the data for a 
better understanding of the overall component score and to provide the component 
program manager with insight into the individual test attributes that may serve as means 
to identify areas that can be addressed to improve component interoperability. 



Figure 13. “Auto IQ” Web Portal. 

The Auto IQ Web Portal provides a convenient method to display SoS assessment results 
and a drill down capability to see how the components of a SoS contributed to the overall 

IQ score (From; Lawver, 2007). 

From a MAGTF C4 SoS assessment perspective, the IQ methodology provides a 
similar approach to the MC3 risk-based reporting methodology with performance data 
obtained from the individual systems during execution of the test thread used to develop a 
measure of risk associated with the success of executing a test thread in the SoS. 
However, the IQ test model examines both internal and external interfaces through 
performance metrics that once defined, are intended to provide a quantifiable and 
repeatable metric that characterizes the overall interoperability of the SoS. 
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Dynamic Software Architecture Visualization and Evaluation 
(DynSAVE) 


While not specifically offering a methodology that leads to an interoperability 
score, the Fraunhofer Center for Experimental Software Engineering Maryland (EC-MD) 
provides an approach for assessing the key interoperability aspects of digital information 
exchange. The EC-MD is a not for profit, applied research organization associated with 
University of Maryland that examines advanced software engineering techniques and 
conducts “applied research in software architecture, verification & validation, process 
improvement and measurement” (Ganesan, Eindvall, Bartholomew, Blau, McComas, & 
Cammarata, 2008, p. 4). To address the issues with performance of complex software 
SoS, the EC-MD developed an approach for assessing software SoS performance and 
identifying problems with communications between systems. The approach was derived 
from an initial effort in support of Johns Hopkins University Applied Physics Eaboratory 
(JHU/APE) Space Department and the development of Mission Operations Center 
(MOC) system software used by NASA missions. The MOC system software used a 
shared architecture called the Common Ground System (CGS) and because of its age (10 
years old), presented challenges when MOC software modification were required in 
support of new mission requirements (Stratton, Sibol, Eindvall, & Costa, 2006). 

To address this challenge, EC-MD in a collaborative effort with the Eraunhofer 
Institute for Experimental Software Engineering (lESE) developed and applied the 
Software Architecture Visualization and Evaluation (SAVE) tool and process: 

This process comprised defining a planned architecture including 
architectural goals and design rationale, generating a high-level 
description of the actual architecture from the legacy software, identifying 
deviations between the planned and actual architecture, creating a new 
target architecture, and creating a roadmap to align on-going systems 
development and maintenance with new target architecture. (Stratton et 
ah, 2006, p. 1) 

The SAVE tool provides a means to automatically extract architectural views 
from the application’s source code and checks the compliance of source code with the 
planned architecture (Ganesan et ah, 2008) as depicted in Eigure 14. 
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Figure 14. SAVE Tool Analysis. 

The SAVE tool, provides a means to automatieally extraet arehitectural views from the 
application’s source code for comparison with the planned architect (From; Ganesan et 

ah, 2008, p. 9). 


Primarily applicable to single software applications, SAVE conducts static 
analysis of software systems written in a variety of languages (C/C++, Java, Delphi, 
Simulink, and others) to identify variances (violations of interaction between 
components) in planned versus implemented software architecture (Stratton et ah, 2006). 

In a follow-on effort, FC-MD extended the functionality of SAVE to provide an 
ability to conduct dynamic analysis of a software SoS. The new product called 
DynSAVE (Dynamic SAVE) provides both structural and behavioral architecture 
analysis of a SoS. Asserting that the reliability of inter-systems communication 
determines the success of the overall capability of a SoS, DynSAVE provides an 
approach for capturing, processing, and visually representing communications behavior 
of a SoS for analysis (Stratton, Sibol, Lindvall, Ackermann & Godfrey, 2009). 
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The challenge associated with inter-systems communications typically stem from 
individual systems implementation of communications protocols. While the protocols 
are typically defined in the system’s Interface Control Document (ICD), a number of 
factors influence the implementation of protocols that can result in degraded SoS 
performance: 

1. The systems are often developed independently from each other by 
different development teams at different locations. 

2. The communication behavior of each individual system is not and 
cannot be fully tested in the environment it will eventually operate 
in, but rather by using components that simulate the 
communication of other systems. 

3. The ICD that specifies the protocol is ambiguous as it omits details 
allowing developers to interpret the protocol differently. 

4. The ICD consists of hundreds of pages of text written in natural 
language, making it difficult for developers to fully understand and 
implement. In the case of CFDP, most implementations only 
support the commonly used features of the protocol, so issues with 
integration of differing subset implementations emerge. 

5. Many clients exist that implement the ICD protocol and it would 
be a significant effort to update them if the protocol was changed. 

6 . Violations of the protocol are not clearly visible but manifest 
themselves as some kind of misbehavior. (Stratton et ah, 2009, p. 

3) 

Subtle difference in protocol implementation can result in anomalies in 
communications behavior that impact reliability and efficiency of the SoS. The 
DynSAVE methodology classifies the anomalies in terms of Sequence (defining the order 
of interaction between systems during message exchange). Parameters (control signals 
and data within a message that controls system behavior) and Timing (time constraints of 
message exchange between systems). The DynSAVE approach depicted in Eigure 15 
involves capturing raw data during communication exchanges, mapping the raw data to 
protocol, and then visualizing the communications behavior in a sequence diagram for 
interpretation and evaluation of the message exchange between systems within a SoS. 
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Figure 15. DynSAVE Process. 

The DynSAVE Process captures raw data for interpretation and visualization in a 
sequence diagram (After; Stratton et al., 2009, p. 6). 


While the MC3 MAGTF C4 SoS assessment approach has looked at capturing 
emergent or unexpected behavior within the SoS during execution of a SoS performance 
assessment in a qualitative manner, DynSAVE may provide a means for applying a more 
quantitative measure to anomalous behavior. Although limited to IP-based protocols, 
DynSAVE can provide greater insight into the origins of anomalous behaviors in a 
complex SoS in terms of the sequence, parameters and timing attributes of digital 
communications; offering a means of quantifying the anomalous behavior and related 
measure of efficiency and reliability of a SoS through those attributes. 
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C. SUMMARY 

A key and certainly foundational aspect of the C4 SoS is the myriad of software 
systems that process the data necessary to provide the C2 capability. The interoperability 
assessment of both i-Score and IQ address this with a specific focus on interfaces and 
data exchange between components that are primarily software systems or software 
systems within a functional component of the SoS (e.g. AFATDS in the DASC 
component). As depicted in Table 3, the i-Score methodology examines the interface 
from a macroscopic level assigning a value to the relative maturity level of the digital 
interface between components. The IQ methodology goes a step further suggesting that 
both interface between components and internal processing within a component are 
necessary to derive an overall SoS interoperability metric. However examining the 
internal logic and boundaries of the components within a complex SoS is not trivial. To 
address this complexity, the FC-MD DynSAVE tool and process provides an approach 
for identifying and visualizing digital communications’ anomalies to convey a better 
understanding of the information exchange attributes that influence both the efficiency 
and reliability aspects of a SoS. 
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Assessment 

Attributes and Applicability to 


Limitations 

Tool 

MAGTF C4 SoS Assessment 



i-Score 

• Quantitative approach for measuring 

• 

Accommodation of parallel 


SoS interoperability through metric 


processes in test thread 


associated with interfaces between 

components 

• 

No direct correlation between 

interoperability score and SoS 


• May provide means for selecting test 


performance metrics 


thread based on i-Score metric and 




relative interoperability 




• May provide a means to quantify 




deviation from intended thread execution 




during an assessment 



IQ 

• Quantitative approach for measuring SoS 

• 

Interoperability scoring 


interoperability through metric 


methodology relies heavily on 


associated with interfaces between 


subjective scoring and 


components and processing within a 


weighting of interoperability 


component. 


attributes 


• May provide a methodology for more 

• 

Summation methodology only 


clearly defining SoS assessment metrics 


valid if scoring is based on a 


• May provide a means to better visualize 


ratio scale 


how lower level metrics contribute to 

• 

No direct correlation between 


overall SoS assessment metric 


interoperability score and SoS 

performance metrics 

DynSAVE 

• Tool and process for identifying and 

• 

Applicable to software C4 SoS 


visualizing anomalies associated with 


transactions only (does not 


digital communications within a SoS 


accommodate non-digital 


• May provide a means to quantify SoS 

anomalies and aspects of efficiency and 

reliability 


transactions and components 

that may contribute to the 

efficiency and reliability of the 

SoS as a whole) 


Tables. Assessment Tool Summary. 

Attributes, applieation and limitations of three SoS assessment tools. 
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While the tools deseribed in this seetion all offer unique methodologies for 
assessing SoS performanee that may help to better quantify the performanee of a 
MAGTF C4 SoS, they are also very similar in that they each examine information 
exchanges at the component level to derive an overall SoS performance assessment 
measure in terms of interoperability. In the next section, multicriteria identification is 
examined as a means to accommodate not only the interoperability measure of a SoS but 
other attributes as well for a more holistic and complete assessment of the SoS. 
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IV. IMPROVING THE SOS MODEL THROUGH 
MULTICRITERIA IDENTIEICATION 


A. INTRODUCTION 

The JMETC Joint SoS testing approaeh and MCTSSA C4 SoS assessment 
approaeh share similar features. For both, execution of a mission-based test thread 
provides the stimulus for the components of the SoS and serves a basis for providing an 
indication of SoS suitability (ability to complete the test thread). For both, the mission- 
based test thread is executed in a SoS that is to varying degrees, an abstract model of the 
actual SoS employed by the end-user (FVC concept). The use of Modeling and 
Simulation (M&S) is not only necessary from a practical standpoint, but also aligned with 
the guidance provided in the 2008 Systems Engineering Guide for Systems of Systems: 

Models, when implemented in an integrated analytical framework, can be 
an effective means of understanding the complex and emergent behavior 
of systems that interact with each other. They can provide an environment 
to help the SoS SE team to create a new capability from existing systems 
and consider integration issues that can have a direct effect on the 
operational user. 

Because it can be difficult or infeasible to completely test and evaluate 
capabilities of the SoS, M&S can be very effectively applied to support 
test and evaluation at different stages throughout the SoS SE process. In 
particular the SoS SE team should consider M&S of the SoS to understand 
the end-to-end performance of the overall SoS prior to implementation. In 
some cases it is advisable for the SoS SE team to adopt a model-based 
process. (OSD, 2008, p. 10) 

In another level of abstraction beyond the physical models that are instantiated as 
the basis of SoS testing and assessment by JMETC and MCTSSA, the construction and 
use of mathematical models that describe the behavior of a SoS would also seem 
relevant. A mathematical model would provide a means for assessing behavioral 
characteristics resulting from changes to key SoS attributes in pursuit of understanding 
how and where improvements to the actual SoS can be made. However, developing a 
mathematical model that represents the complex behavior of a SoS is a significant 
challenge. Furthermore, even with that challenge addressed, obtaining an understanding 
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of how adequate or how elosely the model ean prediet aetual behavior is neeessary before 
that model ean be extended and used for any deeision analysis. 

A set of mathematieal equations that deseribe the operation or performanee of a 
system is fundamental to optimization problems. The equation variables and eonstraints 
assoeiated with the equations define the design spaee and serve as the basis of analysis in 
pursuit of optimizing performanee eriteria. Looking at this proeess in reverse to 
determine and improve a mathematieal model from observed operation and performanee 
of a system is the basis of the identifieation proeess. Multieriteria identifieation provides 
an approaeh for determining the adequaey of a mathematieal model relative to the aetual 
behavior of the system or SoS that the model represents. Onee the mathematieal model is 
determined adequate, it ean then be used for improving physieal models or prototypes. In 
diseussion of establishing adequaey through various identifieation methods, Statnikov 
and Matusov (2002) offer this perspeetive: 

In the most eommon usage, the term “identifieation” means eonstruetion 
of the mathematieal model of a system and determination of the 
parameters (design variables) of the model by using the information about 
the system response to known external disturbanees. In a sense, 
identifieation problems are inverse with respeet to optimization problems. 

(p. 88) 

From a SoS perspeetive, the mathematieal model would ideally represent performanee at 
the SoS level with determination of the parameters of the model obtained during 
exeeution of a mission task (external disturbanee). The feasibility of eonstrueting a 
mathematical model to represent the MAGTF C4 SoS and conceptual use of multicriteria 
identification for establishing and improving the test environment as well as adapting 
elements of the multicriteria identification process as part of the SoS assessment process 
itself are examined below. 

B. MAGTF C4 SYSTEM OF SYSTEMS PROTOTYPE 

The representative MAGTF C4 SoS that serves as the foundation of the 
assessment process consists of computers, databases and applications that receive, 
manipulate, send and display data over a communications and computer network 
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infrastructure for use by machines or humans during the proseeution of a mission thread. 
While often eonsisting of many of the same components found in the Marine Corps 
Operating Foree’s C4 SoS, the representative C4 SoS in the test environment is usually 
an abstraction of the actual environment. 

The test environment is a physieal model of the aetual SoS and may also be 
eonsidered a prototype that ean serve as an environment for testing new C2 applications, 
modifying system eonfigurations or modifying communications protocols or networking 
attributes that we hope will improve, or at least do no harm to the actual C4 SoS. As we 
modify the prototype, the hope is the resulting performanee, interoperability measure or 
other attributes of the SoS will adequately reflect what we would expect in the aetual 
SoS. A eomplementary mathematieal model that deseribes the behavior of the SoS 
would provide a means to improve the physical model or prototype through the 
multicriteria identifieation process. 

C. MAGTF C4 SYSTEM OF SYSTEMS MATHEMATICAL MODEL 

Mathematieal modeling of C2 eommunieations and network architectures is 
pursued for a variety of uses. One of the most widely known modeling eonstructs used 
within the Department of Defense (DoD) is OPNET Modeler. This modeling eonstruct 
focuses primarily on communications systems and provides a eapability for 
eommunieations systems planning and performance predietion. Wargaming simulations 
like JANUS(T) use mathematical models that serve as the foundation for determining 
predictive outeome during military operational scenarios. These models focus on 
performance of military units with inherent C2 capabilities that can infer C2 performance 
(Ingber, Fujio, & Wehner, 2001). 

Developing a mathematical model that deseribes the behavior of the MAGTF C4 
SoS hints at the very essenee of the SoS assessment challenge: defining the key SoS 
performance eriteria. The key performanee eriteria that serve as the basis of an 
assessment are eentral to the eonstruetion of a mathematical model that describes the 
SoS. The i-Score and IQ methodologies quantify the SoS by applying the prineiples of 
decomposition and aggregation of large-seale systems with an analysis and a seoring 


57 



construct that defines an interoperability measure as the key performanee criteria. 
Measuring interoperability at the individual system level is an approaeh shared by 
i-Score, IQ and DynSAVE. Improving the interoperability between eomponents through 
i-Seore, IQ or DynSAVE analysis would imply improved effieiency (reduetion in human 
or maehine proeessing requirements within the SoS as the thread is exeeuted) and 
improved effeetiveness (reduetion in proeessing implies potential for quieker exeeution 
of the thread with less potential for introduetion of error during translation from one 
eomponent to the next). These approaehes eaeh provide a means for defining and 
measuring attributes at the systems level for aggregation into a meaningful overall 
interoperability measure that should eorrelate to the overall effieiency and effectiveness 
of the SoS. 

While IQ and DynSave provide a means of deriving a measure of interoperability 
through observed and quantitative assessment of a SoS, only the i-Seore methodology 
provides an approaeh to deseribing a SoS in terms of a mathematieal model. The i-Seore 
approaeh to modeling a SoS in terms of interoperability is not unique. Other approaches 
have been proposed to include a model that deseribes the SoS in terms of interoperability 
and eomplexity presented in A Model for Assessing the Performance of Interoperable, 
Complex Systems by Thomas Huynh and John Osmundson (2006). However, the use of a 
mathematieal representation of a SoS in terms of interoperability is limited unless we ean 
somehow eorrelate the interoperability measure with measurable performanee eriteria of 
the SoS. To address this eommon ehallenge, prototype experimentation may provide a 
means to help determine the eorrelation between an interoperability measure and 
performanee of the SoS and then help improve the model through parametric 
identification (improving or determining the appropriate numerieal values of the equation 
eoefficients). 

1. Prototype Experimentation Data to Develop and Improve Model 

In a broader perspeetive, we would like to charaeterize the C4 SoS in terms of 
overall C2 effeetiveness (ability to effieiently and effeetively provide a C2 eapability) as 
the main eriterion. The SoS should provide a C2 eapability while eonsuming as few 
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resources as possible (measure of efficiency) while enabling the successful execution of a 
mission thread as quickly as possible (measure of effectiveness). The interoperability 
measure addresses this in part, but with focus solely on thread execution and 
interoperability between systems, other attributes of the C4 SoS that are necessary to 
provide indirect or background C2 activities may not be adequately accommodated. 
Activities such as position reporting, DISN services, and maintaining Situational 
Awareness (SA) may contribute directly to the execution of a mission thread, but are also 
critical background processes that enable decision making to ensure the appropriate 
mission is executed at the appropriate time for successful execution of a warfighting 
campaign. From that perspective, a mathematical model would need to accommodate 
those activities as variables within the model (demand for services varies with the 
operational tempo and battle-rhythm that is driven by the operational scenario and 
echelon). Those variables as well as the activities directly engaged with execution of the 
mission thread are consumers of bandwidth and the quality of service related to those 
activities is dependent on bandwidth availability. Availability of bandwidth is generally 
a fixed constraint within the SoS although allocation of bandwidth within the SoS to 
specific C2 services can be optimized to best meet operational needs. Ideally, although 
beyond the scope of this study, a mathematical interoperability model would be 
correlated with measurable attributes of C2 effectiveness (such as bandwidth efficiency, 
resource usage and time to execute mission tasks), while accommodating the variable 
background activities and fixed bandwidth constraint. 

a. Simulation and Stimulation Considerations 

To accommodate the C2 background activities in the model, some thought 
must be given to how they can be represented in the prototype. The operational scenario 
serves as a reference for the mission-based test thread and drives the demand for C2 
services. However, the demand for C2 background activities is largely dependent on the 
battle rhythm and operational tempo during mission execution and not necessarily 
dependent on mission execution itself Real-world operational data can provide a basis 
for determining the variations in C2 background activities (Position Location Information 

or track load and DISN services such as VTC, telephone, and collaboration services) that 
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can be replicated in the C4 SoS prototype thorough various simulation and stimulation 
applications. Executing the test thread in the prototype environment under various mixes 
and load of C2 background activities will induce more realistic behavior necessary for 
developing a higher fidelity mathematical model. 

The use of virtualization technology may also offer a more efficient means 
to establish a large-scale C4 SoS model. MCTSSA experimented with using 
virtualization technology to replicate a MAGTF C4 SoS during the 2008 MC3 
assessment effort. Virtualization technology provided a means to rapidly construct the 
C4 architecture to validate individual component configuration settings and test 
procedures to be used during the assessment. Once configuration settings and test 
procedures were validated, the C4 architecture was constructed with actual C4 
components for the assessment. This resulted in considerable time savings (2-3 days) 
over previous MC3 events by allowing the test procedure validation effort to be 
conducted in parallel with other pre-assessment preparation efforts. Use of a virtualized 
environment to conduct the actual assessment was not considered due to the unknown 
differences in performance between the virtual and actual C4 SoS models (Marine Corps 
Tactical Systems Support Activity, 2009). 

b. Data Collection Considerations 

Performance data related to C2 effectiveness can be obtained from a 
measure of the time to execute the mission thread. Performance data related to C2 
efficiency may require use of tools to measure bandwidth utilization as a measure of 
efficiency or new tools like DynSAVE that capture the behavior of complex SoS data 
transactions for more detailed efficiency analysis. 

In an operational C4 SoS, network optimization and Quality of Service 
(QoS) management are performed through use of commercial software tools such as 
Solar Winds’ Orion Network Performance Monitor associated with the Joint Network 
Management System (JNMS) program of record, Cisco’s suite of network management 
products associated with the TDN program of record, and Referentia’s EiveAction 
network management solution associated with an Office of Naval Research (ONR) 
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development effort. While these tools serve as an integral part of the SoS, they are not 
always ineluded in the test/assessment prototype environment since they do not directly 
contribute to the execution of a mission thread. However, if C2 background activities are 
incorporated within the SoS prototype, these tools are necessary to ensure the network is 
configured to reflect realistic operational requirements with respect to QoS and network 
performance optimization. Further, while these tools are an element within the SoS, one 
of their key functions is to provide internal diagnostics relative to the SoS network 
performance and while considered intrusive from a test environment data collection 
perspective, may still provide critical insight into and measurement of performance 
efficiency from a resource utilization perspective. 

2. Multicriteria Identification 

The complexity of a MAGTF C4 SoS may prohibit development of an adequate 
mathematical model in terms of a single performance criteria. Bandwidth efficiency, 
resource usage and time to execute mission tasks are three criteria identified earlier, but 
there could be many other performance criteria that must be considered to adequately 
describe the SoS—that is to say, a mathematical model that reflects the correlation 
between the interoperability measure and bandwidth efficiency performance measure 
may differ from the mathematical model that reflects the correlation between the 
interoperability measure and time to execute a mission task. Yet, both may be needed to 
adequately describe the SoS. This would indicate that correlating interoperability with 
SoS performance may very well be a multicriteria problem. Developing and improving 
the mathematical relationship between interoperability and SoS performance then would 
involve a multicriteria identification process: observing performance of the prototype 
during mission-based thread execution and through experimentation develop and improve 
the mathematical relationship. 

a. Use of Adequacy Criteria to Improve the Models 

Adequacy criteria (or proximity criteria) represent the discrepancy 
between the physical prototype model and the mathematical model (Statnikov & 
Matusov, 2002). Once a mathematical model is developed that correlates interoperability 

61 



with the performance criteria, the model can be improved through multicriteria 
identification to a level of adequacy that is deemed acceptable. Statnikov & Matusov 
(2002) define the experimental value of the v*’’ criterion measured from the prototype as 
and the calculated value from the mathematical model as to derive an 
expression for the adequacy criterion as follows: 

I (4.1) 

For our purpose, the adequacy criteria would be a measure of how well 
correlated the performance criterion determined by the interoperability model is with the 
actual measurement from the prototype model. Through considerable experimentation 
with the prototype and an understanding of the nature of the measurement error 
associated with the performance criterion of the prototype, a maximum value of the 
adequacy criteria can be determined through expert analysis and in notation presented by 
Statnikov and Mutusov (2002), written as an inequality: 

I 0/-0/’^P|<Sv. (4.2) 

The value of Sv reflects the maximum value of the adequacy criteria 
(desired accuracy of correlation between physical and mathematical models). Examining 
the variables associated with the models that still satisfy the inequality will ensure that 
alternative representations of the models are identified that may prove useful specifically 
during efforts to optimize the performance criteria. With a mature mathematical model 
and high degree of confidence in the correlation between the performance criteria 
measure predicted by the interoperability model and the performance measure observed 
from the prototype, the mathematical model may provide a cost effective and rapid means 
to examine the SoS for potential performance improvement. 

b. Use of Adequacy Criteria as Baseline Measure for C4 SoS 
Performance Assessment 

The process required to develop and mature a mathematical model of the 
SoS also provides greater insight into the associated performance metrics. With an 
understanding of the nature of the measurement error associated with the prototype, 
expected values for the performance metrics could be determined and used as a basis for 

62 



assessment. The inequality (4.2) eould serve as a suceess criteria that must be satisfied 
during a SoS assessment. If an assessment results in not meeting the criteria (I - 
I > Sy), further evaluation of the SoS would be required to determine if a change to 
the SoS invalidated the mathematical model or if the SoS just failed to perform as 
expected. 

D. SUMMARY 

Modeling the MAGTF C4 SoS is an important facet of the assessment effort. A 
representative MAGTF C4 SoS that serves as the assessment environment is an 
abstraction of the actual operational SoS environment and from that perspective can be 
considered a prototype. This prototype can serve as a means for assessing the overall C2 
effectiveness of the SoS if appropriate simulation, stimulation and data collection 
activities are employed to accommodate both mission thread execution and critical C2 
background activities. A complimentary mathematical model would also add value by 
providing a capability to independently predict SoS performance behavior and gain a 
better understanding of where and how to best optimize the prototype environment in 
pursuit of developing a more effective SoS in the operational environment. The i-Score 
methodology provides a mathematical model that relates an interoperability score to the 
physical attributes of the SoS, however the interoperability score would be significantly 
more useful if it can be correlated with performance criteria of the SoS. The use of 
multicriteria identification may provide a way of modifying the i-Score model (or other 
mathematical interoperability model) to establish a correlation between the 
interoperability score and relevant performance criteria. If developed, the mathematical 
model could provide an efficient means to examine modifications to the SoS in pursuit of 
performance improvement. Additionally, the use of adequacy criteria determined 
through the multicriteria identification process may serve as a more reasonable 
benchmark for evaluating SoS performance during an assessment event by providing an 
expected value for a performance criteria that accommodates both statistical variance in 
SoS performance behavior and aspects of known measurement error. 
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V. CONCLUSIONS 


A. KEY POINTS AND RECOMMENDATIONS 

Over a number of years, the Marine Corps Taetieal Systems Support Activity has 
aggressively pursued new and innovative ways to improve efforts to assess the 
performance of tactical C4 Systems of Systems prior to deployment in an operational 
environment. In that pursuit, determining how best to approach performance assessments 
of large-scale, complex SoS has proven a challenge that extends beyond the Marine 
Corps to a fundamental system of systems engineering challenges shared by acquisition 
and test organizations and activities throughout the DoD. Selecting key performance 
criteria, developing methodologies for conducting large-scale SoS assessments and 
defining more quantitative means for measuring and assessing SoS performance and 
behavior all serve as a central challenge and the genesis of the initial questions that 
guided this research: 

1. How can large scale C4 System of Systems be tested and evaluated in a 
manner that reflects performance attributes associated with the System of 
Systems as a whole? 

2. What are the key attributes of a C4 SoS that may serve as the basis for SoS 
level performance criteria? 

3. What are some existing assessment tools and how may they be extended to 
aid in C4 SoS assessment process? 

4. How might multicriteria identification improve or contribute to the Marine 
Corps’ C4 SoS assessment methodology? 

For Joint SoS performance assessments, the use of mission-based test threads 

serve as both a means of stimulating the SoS and a means for assessing performance at 

the SoS level (capability of the SoS to execute the mission thread from a timeliness and 

completion percentage rate). The use of a mission-based test thread is also extended to 

MAGTF C4 SoS assessment efforts, but with a specific C4 focus, other attributes 

associated with providing a C2 capability are also of interest. Incorporating C2 
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background activities (situational awareness, position reporting, collaboration, and other 
DISN serviees) within the modeled C4 architecture provides an ability to assess aspects 
of the SoS with respect to providing a C2 capability (capability of the SoS to provide C2 
services with regard to bandwidth efficiency and network reliability). 

The use of network management and optimization tools that are typically key 
eomponents of operational C4 SoS have not typically been ineluded in MAGTF C4 SoS 
assessment events since they do not directly contribute to exeeution of a mission thread. 
Adding these components to the assessment environment provides a means to better 
model QoS and bandwidth availability attributes assoeiated with the SoS and may 
provide additional insight into and measurement of performance effieiency from a C2 
background activity and resource utilization perspective. 

In reference to the first two questions that guided this researeh then, it is 
reeommended that MAGTF C4 SoS assessment efforts include aspects of providing a C2 
capability in addition to the ability of the SoS to execute a mission-based test thread as 
the basis for selection of SoS level performance criteria. This, in concert with addition of 
network management and optimization tools, will provide the foundation for an 
assessment with foeus on the key attributes of a C4 SoS from a more holistie, SoS 
performanee perspective. 

A measure of SoS interoperability also provides a useful means of eharacterizing 
a C4 SoS in terms of information exchange attributes that influence both the efficieney 
and reliability of a SoS. Guided by the third question of this study, three assessment 
tools were reviewed with eaeh providing a unique approach for evaluating and measuring 
SoS interoperability. The most promising tools for continued study are the i-Seore 
methodology with an approaeh that provides a quantitative measure of SoS 
interoperability, and the DynSAVE software tool that provides a means of quantifying 
the anomalous behavior and relative effieieney of the SoS. Based on this, demonstrating 
and further evaluating the improved i-Score methodology is reeommended to determine 
if the higher level of fidelity offered by the improvements add to or enhanee the 
applicability to a C4 SoS assessment event. Demonstrating and evaluating DynSAVE is 

also reeommended to determine whether the structural and behavioral architeeture 
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analysis provided by DynSAVE can be extended to provide a measure of efficiency and 
reliability during a MAGTF C4 SoS assessment. DynSAVE may also illustrate specific 
data exchange attributes between SoS components that may help determine or improve 
the resemblance coefficients necessary for the improved i-Score model. 

A mathematical model that determines a quantitative value for SoS 
interoperability like that offered by i-Score may certainly complement a physical model 
of the SoS like that used for MAGTE C4 SoS assessments. However, the interoperability 
measure would be considerably more useful if it can be correlated with SoS performance 
criteria. Guided by the fourth question of this research, multicriteria identification can 
contribute to the Marine Corps’ C4 SoS assessment methodology by providing a means 
to modify the i-Score model (or other mathematical interoperability model) to establish 
that correlation. With a mature mathematical model and high degree of confidence in the 
correlation between the performance criteria measure predicted by the interoperability 
model and the performance measure observed from the prototype, the mathematical 
model may provide a cost effective and rapid means to examine the SoS for potential 
performance improvement. Additionally, the use of adequacy criteria determined 
through the multicriteria identification process may serve as a more reasonable 
benchmark for evaluating SoS performance during an assessment event. Eor those 
reasons, pursuing development of a C4 SoS mathematical model is highly recommended. 

Einally, while SoS assessment is a significant challenge, it is a challenge that is 
shared across the DoD and throughout the systems engineering community. Continued 
involvement in Joint DoD efforts such as JMETC provide the Marine Corps with a 
unique opportunity to refine assessment methodologies, examine new tools and 
techniques and leverage expertise and lessons learned through continued collaboration 
within this growing community of interest. Engagement is not just a good idea, but is 
essential to improve and mature the Marine Corps’ engineering and test effort as applied 
to current and future generations of the MAGTE C4 SoS. 
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B. AREAS TO CONDUCT FURTHER RESEARCH 


A number of areas diseussed in this study warrant further examination. First, 
demonstration and evaluation of the improved i-Seore methodology may be benefieial to 
determine if the improved methodology shows potential for greater applieability to C4 
SoS assessments. Seeond, demonstration of the DynSAVE assessment tool during a C4 
SoS assessment would provide an opportunity to examine eapabilities and limitations of 
this tool in a eontrolled environment. Third, further study towards development and use 
of a mathematieal model for C4 SoS assessment and optimization may provide 
eonsiderable benefit. While extensive experimentation through a physieal model of the 
C4 SoS to develop and improve the mathematieal model is neeessary, applieation of 
simulation and stimulation tools and virtualization teehnology may help expedite this 
effort. Finally, virtualization teehnology may serve as a fruitful area of study from two 
perspectives: using a virtual model of the SoS as the basis for an assessment, and 
determining appropriate tools and analysis methodologies for assessing C4 SoS that 
employ virtualization technology as part of their inherent architecture. 

1. C4 SoS Assessment Using a Virtual Model 

The use of virtualization technology to model the C4 SoS offers considerable 
advantage by reducing the physical resources needed to create the C4 SoS assessment 
environment. The benefits of using a virtual model to validate assessment procedures has 
been demonstrated, but even greater value would be derived if a virtual model of the C4 
SoS could be extended to serve as the actual assessment environment. Because the 
virtual model presents another degree of abstraction from the physical model, 
experimentation and analysis is necessary to determine where and how variances between 
the virtual and physical models manifest in terms of SoS performance measures. 

2. Assessment Tools and Analysis Concepts for C4 SoS Analysis in 
Virtual SoS Analysis Environment 

A number of Marine Corps’ Systems of Systems, including the COC, intend to 
employ virtualization technology within their architectures. Virtualization of network 
components (servers and switches), as well as various C2 client components, will change 

68 



the performance and behavior attributes of the SoS. Additional research is needed to 
determine whether current assessment tools, techniques and performance criteria are 
sufficient to conduct an assessment of a C4 SoS that employs virtualization technology or 
if additional or new tools are necessary. 
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